CS 6983 Special Topics: The Technology of Lambda		-*- Outline -*-
Olin Shivers

Project 5: Closure conversion -- Analysis 3: Stack layout
Due 2026/3/31 midnight by git commit

* 1. Project description
---------------------
In the final analysis (pun intended), it's necessary to walk the code
tree and determine the layout of each function's stack frame. This
analysis has to come last, as it depends on a lot of the information
your prior analysis passes have calculated and left as annotations
on the various nodes of the AST.

This project description also has a summary at the end providing
a complete list of all the annotations your compiler's various
analysis passes add to the original CPS intermediate representation,
in all the (@ <annotation> ...) forms scattered across the AST. Make
sure you understand the *meaning* of each one -- in the next phase
of the compiler, your code will use these annotations to translate
the CPS down to the lower-level "machine code" form.

-------------------------------------------------------------------------------
* 2. Frame layout
-----------------
The last thing we have to do before transforming our code is to lay
out our stack frames. Our implementation will use a stack in more
or less the same way that C code does. Depending on how you look at
it, a stack frame is either
- Where we close CONT lambdas (that is, put the values of their
  free variables), or
- where we save variables that are live across function calls.
(It's the same thing.)

In this analysis, you will lay out the stack frames. These decisions
will be recorded in some new annotations.

Let's begin by defining our stack management policy.

We say a CONT form "belongs" to the innermost first-order or closure
user-lambda FUN that contains it.
- Containing Open FUNs don't count, as we don't allocate a new stack
  frame on entry to one.
- Containing CONT forms don't count, either, for the same reason.

Our model is that

- Whenever control enters a user lambda (a FUN), we allocate (push)
  a fresh stack frame: sp -= <frame-size>.

- Whenever a CONT closure is created, it is allocated on / inside that
  user-lambda's stack frame -- which is the one at the top of the stack.

  That is, we pick some unused slots on the current stack frame and
  save the CONT's free variables there; the CONT code is then compiled
  so that its free-variable refs are rendered as loads from the stack.
  It's the job of the CONT's caller (the function that returns to this
  continuation, that is) to pop the stack back to this frame before
  jumping to the CONT's code. Just as a heap-closure FUN loads its
  free variables from its closure record, the stack-closed CONT loads
  its free variables from the stack frame.

- When a user lambda returns (whatever that means, precisely -- see
  below), its frame is popped: sp += <frame-size>.

What precisely do we mean by "when a user lambda returns"? We mean one
of three possibilities. Suppose we're talking about user lambda
    L = (fun (x k) ...)
and L is a frame-allocating lambda (that is, 1st-order or closure, not
open). Then we pop L's frame when

1. The code does a simple return, by which we mean, when it calls K:
     (K 36) ; Return 36 to L's caller.

2. The code does a tail call, of which there are two kinds:
   2a. A user-function call with continuation K:
         (F 7 K) ; Hand off to F, who will return answer to L's caller.
       This gets at the idea that a tail-call is a pop-then-call --
       We no longer need L's frame, and F should just return to L's
       caller, K.
       (This exploits a subtle invariant: K's frame is always the one
       *just before* the current L frame on the stack, so popping the
       current frame will get us back to K's frame.)
   2b. A primop call with continuation K:
         ($* Y 3 K) ; Return Y * 3 to L's caller.
       This is simply: do some primitive piece of work (multiply,
       in this example), and *then* return to K. So it's a return.

The sign of a return is L's continuation parameter. When we talk about
buried treasure, X marks the spot. But when it comes to returning and
popping a frame, it's not X: K marks the spot.

Again, the big idea is that when control enters a (closure or
1st-order) FUN, its prelude code bumps the stack *once* and allocates
a frame large enough to serve the closure needs of *all* the closure
continuations that "belong" to this user lambda.

Note that the CONT forms belonging to user lambda L make a tree rooted
at L, linked by PARENT-BINDER up-link annotations. To find the FUN
that "owns" CONT C, just follow these up-links through intermediate
containing LETREC and CONT forms until we get to a FUN form.

How big is this frame? Your frame analysis will leave a FRAME-SIZE
annotation on the FUN term to say:

    (fun (x y k)
      (@ ... (frame-size 5) ...)	; Need 5 slots (slots, not bytes)
      <body>)

We also have a new annotation for closure CONT forms, that says what
its incremental needs are, the FRAME+ annotation, which looks like
this:

    (cont (x y)
      (@ ...
         (frame+ (<var> <slot-num>) ...)
         ...)
      <body>)

The FRAME+ annotation specifies the variables that should be added to
the current stack frame for this continuation's needs when it runs.
Put another way: it is the set of (extended) free variables this
continuation needs in its closure (i.e., on the current stack frame)
that haven't *already* been put on the frame by some previous CONT
form appearing in-between this one and the user-lambda FUN to which
this one belongs. Let's call that chain of nested CONT forms our
"predecessor" CONTs: it is the case that if control has gotten *here*,
where we are evaluating this CONT form, then we've previously
evaluated all of our predecessors, so all of the variables *they* put
on the stack are *still there*. We don't need to save these variables
a second time. We just need to save our *additional* needs.

Let FP (for "frame plus") be the set of variables in a CONT's FRAME+
annotation. Let XF be the set of variables in a CONT's EXTENDED
free-variable set. Our CONT is going to need all the variables in
set XF saved on the stack -- but all the variables in set XF - FP
are *already there*. We only need to save the variables in FP.

We can find the FRAME+ vars for a continuation with a tree walk. As we
walk the AST top-down, we'll keep track of CF, the "current frame"
layout, which says what variables are saved on the stack... and where.
So if CF = {x |-> 2, z |-> 5}, then the variable X is at slot #2, and
Z is at slot #5 on the stack frame -- and slots 0, 1, 3, & 4 are
currently unused.

- A stack slot "var |-> num" dies and is removed from the stack map CF
  when we descend into a subtree of the AST that has no free reference
  to var. We no longer need the variable's binding, so this reclaims
  the slot for further use.

- When the compiler recurs down into a 1st-order or closure FUN, we
  reset the frame map to the empty map {} -- we're starting a fresh
  frame, which will be laid out as the compiler walks the body of the
  FUN. (Open FUNs, by contrast, have no effect on the state of the
  stack.)

- When we descend into a closure continuation C = (CONT ...), with
  frame map CF, we strip out any entries in CF not in C's extended
  free-var set -- as described above, those entries are no longer
  needed, so we can reuse their stack slots. Then we figure out which
  of C's free variables need to be added to the stack:
    XF - Domain(CF)
  That is, the FRAME+ set is all the variables C will need on the
  stack when execution eventually returns to it (that is, XF)...
  except for the variables *already* on the stack at closure time
  (that is, Domain(CF)). That's what the program will need to add to
  the stack when C is evaluated. These are the variables that you'll
  put in C's FRAME+ annotation.

  Of course, every variable in the FRAME+ set will also need to be
  assigned a free slot in the current stack frame. We'll just use the
  lowest slots (the smallest slot numbers) that are unused in CF.

  Put another way, for a given CONT form C, that occurs at a program
  point with current frame map CF, to get the frame map for C's body,

    Remove: Domain(CF) - XF	<-- No longer needed
    Add:    XF - Domain(CF)	<-- This is C's FRAME+ set.

- Then we recur into C's body, using the updated frame map.

- As the analysis returns up through its tree walk of the AST, it
  passes back the maximum frame size needed while laying out the stack
  frame in a child of the current syntax node. The max of these
  high-water marks is the high-water mark for the current node. This
  is how we determine the FRAME-SIZE annotation for a given user
  lambda L: walk its body starting with a fresh, empty frame map CF,
  and use the high water mark returned by the analysis for L's body.

- For purposes of eliminating "fence-post" errors, let's be clear
  about how we count slots, frame sizes and high-water marks. There
  are different conventions possible, but here's a consistent one:
  - Number the stack slots starting with slot 0.
  - FRAME-SIZE is a *size*. A frame size of 0 means no frame needed.
    A frame size of 2 means two slots: slots #0 and #1.
  - So use the same convention for the high-water mark: size, not
    slot index. Thus stack map {x |-> 1, z |-> 4} should produce a
    high-water value of 5, not 4: you need a five-slot frame for
    this layout
        [- | x | - | - | z]
         0   1   2   3   4
    where - marks the three free/unused/dead slots.

A small additional complexity: multiple variables bound by the same
LETREC to closure lambdas (as opposed to first-order lambdas, which
don't actually turn into values at run-time) only need *one* stack
slot. (Because we only save one copy of their shared closure record on
the stack. When we actually have a *reference* to any of these
variables, we'll add the right offset to the record at the reference
point.) You handle this by assigning these variables the *same* slot
in the FRAME+ stack map. For example, in this map
    (frame+ (x 2) (f 4) (y 5) (g 4))
F and G are siblings bound in the same LETREC; what we'll store at
slot #4 in the stack at run time will be a pointer to the shared
closure record that was created for F & G. Whenever we need the actual
value for F or G (to pass as an argument to another function, or
call), we'll load the address of this record off the stack into a
register and then immediately add the constant offset to it that's
needed to make the address point to the slot in the closure record
where F or G's code is stored.

To handle this sharing, you'll have to adjust the dictionaries your
analysis uses for the CF current-frame maps accordingly.

Here's a marked-up example, with comments:
    (cont (x)
      (@ (kind closure)
	 (label c39)
	 (parent-binder fun49)	; C is a first-order var,
	 (free-vars a c q)	;   so it's not in the EXTENDED set.
	 (extended a q v w)	; V, W not used locally; for passing on to C.
	 (frame+ (q 2)		; A & V already on the stack; add Q & W
		 (w 5)))	;   to have all of EXTENDED set on frame.
      <prog>)			; This code runs w/access to all of EXTENDED.

Note that we can always figure out which of a term's free variables
are first order: they are not included in the term's extended free
variables set. So the first-order free variables are
    FV - XF
where FV is the set of free variables for some binder, and XF is the
set of its extended free variables. This is how you can tell that C
is a first-order variable in the above example -- so we know that C
is only *called* in the <prog> body, never used as a value.


-------------------------------------------------------------------------------
* 3. Lambda chart
-----------------
Remember that for every kind of lambda, there are three control points
or times that matter:
- the point where we *evaluate* the lambda
  (which is determined by the code where the lambda appears),
- the point where we *jump to / call* the lambda
  (which is determined very differently for open, first-order
  and closure lambdas), and
- the point when we *execute* the lambda
  (which is the code *of* the lambda -- its parameters & body).

What we do at these points depends on the kind of lambda:

  {open,first-order,closed} x {FUN, CONT}.

And much of this is, in turn, driven by what we know about

1. The relationship between the environment at the point of
   evaluation, and the point of call. E.g., in an open lambda,
   these two environments are identical; in a closed lambda,
   we assume no relationship at all.

2. The relationship between the state of the stack at the point
   of evaluation, and the point of the call. E.g., in a closed
   FUN, there's no useful connection; in a closed CONT, we ensure
   the stack is popped back to the same frame.

Here's a chart. If you can understand the "why" of every entry in all
six boxes, and see what the implementation implications are, you have
attained CPS satori and there is reason to hope that you understand
enough to write the analyses of this assignment correctly.

Maybe.

		FUN				CONT
		---				----
Open		Not used as a value		Not used as a value
		& free vars available @ call	& free vars available @ call
		=> no closure on eval		=> no closure on eval
		=> frame layout inside FUN	=> frame layout inside CONT
		   is same as outside		   is same as outside
		=> No FRAME+ annotation		=> No FRAME+ annotation
		Only action on entry		Only action on entry
		   to bind "reg" params		  to bind "reg" params
		Called immediately		Called immediately
		=> free vars all available	=> free vars all available
		Basically, the var & body	Basically, a LET binding
		  of a LET binding.		  or primop-call continuation.
		Invariant: FUN's cont param
		is closed on / uses *current*
		frame! (Do not pop on call...)

1st-order	Not used as a value		Not used as a value
		& free vars passed from caller	& free vars passed from caller
		=> no closure on eval		=> no closure on eval
		=> eval adds nothing   		=> eval adds nothing
		   to the stack	       		   to the stack
		Entry starts fresh frame!	=> frame layout inside CONT
		=> frame layout in FUN empty!	   is same as outside
		=> No FRAME+ annotation		=> No FRAME+ annotation
		FUN's cont argument is closed
		  on / uses immediately
		  prior stack frame

Closure		Used as value, called from 	Used as value, called from
		  arbitrary lexical context	  arbitrary lexical context
		=> eval makes closure		=> eval makes closure
		   on heap			   using stack frame
		Entry starts fresh frame!	Entry resumes eval-time frame!
		=> No FRAME+ annotation		FRAME+ describes eval-time
						   additions to frame for
		Fun's cont argument is closed	   closure.
		on / uses immediately		Evaluating CONT, making closure
		prior frame.			   is just saving FRAME+ vars
						   on current frame.
						Stack frame when inside CONT
						   is stack frame outside CONT,
		  				   *plus* new FRAME+ slots.


-------------------------------------------------------------------------------
* 4. Annotations list
---------------------
Here is the full set of annotations that are the results of your analyses.

(KIND <kind>)			; lambda - open, first-order or closed
(LABEL <name>)			; binder - unique name
(FIRST-ORDER-VARS var ...)	; binder - first-order vars bound by term
(FREE-VARS var ...)		; binder - free vars referenced by term
(EXTENDED var ...)		; binder - extended free vars
(PARENT-BINDER label)		; binder - innermost containing binder
(CALLERS label ...)		; lambda - parent binder of every call site
(FRAME-SIZE <nat>)		; First-order and closed FUN
(FRAME+ (<var> <slot-num>) ...)	; CONT   - stack saves needed to make closure


-------------------------------------------------------------------------------
* 5. Reflect
------------
Take a look at a fully annotated program produced by your analysis.
Consider *just how much* information now decorates your code -- you
now have everything marked that you'll need to translate the program
down to a machine-level model where there are no lexically scoped,
higher-order functions, only blocks of memory holding data and other
blocks holding code. *All those decisions have been made.*