Lua parser internals

[Dibyendu Majumdar <mobile@...>]

Hi,

As I work on Ravi perhaps one of the most complex areas I have to deal
with is the Lua Parser and Code generator. As I understand more I tend
to add notes for my own benefit - explaining my understanding.

One such note is given below. I would be grateful if Roberto or Luiz
could comment on the correctness of this description.

typedef struct FuncState {
  Proto *f;  /* current function header */
  struct FuncState *prev;  /* enclosing function */
  struct LexState *ls;  /* lexical state */
  struct BlockCnt *bl;  /* chain of current blocks */
  int pc;  /* next position to code (equivalent to 'ncode') */
  int lasttarget;   /* 'label' of last 'jump label' */
  int jpc;  /* list of pending jumps to 'pc' */
  int nk;  /* number of elements in 'k' */
  int np;  /* number of elements in 'p' */
  int firstlocal;  /* index of first local variable (in Dyndata array) -
                      this marks the start of the register
                      window of the function i.e. register
                      zero is here */
  short nlocvars;  /* number of elements in 'f->locvars'
                      - i.e. number of local variables */
  lu_byte nactvar;  /* number of active local variables in
                       the function - see note above on how this works */
  lu_byte nups;  /* number of upvalues */
  lu_byte freereg;  /* first free register - this tracks the top
                       of the stack as parsing progresses */
} FuncState;

/* State needed to generate code for a given function.
   Functions have a register window on the stack.
   The stack is represented in LexState->dyd.actvar (Dyndata)
   structure (see llex.h). The register window of the function
   starts from LexState->dyd.actvar.arr[firstlocal].
   The 'active' local variables of the function extend
   upto LexState->dyd.actvar.arr[nactvar-1].
   Note that when parsing a 'local' declaration statement
   the 'nactvar' is adjusted at the end of the statement
   so that during parsing of the statement the 'nactvar'
   covers locals upto the start of the statement. This means that
   local variables come into scope (become 'active') after
   the local statement ends. However, if the local statement
   defines a function then the variable becomes 'active'
   before the function body is parsed.
   A tricky thing to note is that while 'nactvar' is adjusted
   at the end of the statement - the 'stack' as represented
   by LexState->dyd.actvar.arr is extended to the required
   size as the local variables are created by new_localvar().
   When a function is the topmost function being parsed, the
   registers between LexState->dyd.actvar.arr[nactvar]
   and LexState->dyd.actvar.arr[freereg-1]
   are used by the parser for evaluating expressions -
   i.e. these are part of the local registers available to
   the function
   Note that function parameters are handled in the
   same way as local variables.

   example of what all this means:

   Let's say we are parsing following chunk of code
   function testfunc()
     -- at this stage 'nactvar' is 0 (no active variables)
     -- 'firstlocal' is set to current top of the variables stack
     -- LexState->dyd.actvar.n (i.e. excluding registers used for
expression evaluation)
     -- LexState->dyd.actvar.n = 0 at this stage
     local function tryme()
       -- Since we are inside the local statement and 'tryme' is a
local variable,
       -- the LexState->dyd.actvar.n goes to 1. As this is a function definition
       -- the local variable declaration is deemed to end here, so
'nactvar' for testfunc()
       -- is gets set to 1 (making 'tryme' an active variable).
       -- A new FuncState is created for 'tryme' function.
       -- The new tryme() FunState has 'firstlocal' set to value of
LexState->dyd.actvar.n, i.e., 1
       local i,j = 5,6
       -- After 'i' is parsed, LexState->dyd.actvar.n = 2, but
'nactvar' = 0 for tryme()
       -- After 'j' is parsed, LexState->dyd.actvar.n = 3, but
'nactvar' = 0 for tryme()
       -- Only after the full statement above is parsed, 'nactvar' for
tryme() is set to '2'
       -- This is done by adjustlocalvar().
       return i,j
     end
     -- Here two things happen
     -- Firstly the FuncState for tryme() is popped so that
     -- FuncState for testfunc() is now at top
     -- As part of this popping, leaveblock() calls removevars()
     -- to adjust the LexState->dyd.actvar.n down to 1 where it was
     -- at before parsing the tryme() function body.
     local i, j = tryme()
     -- After 'i' is parsed, LexState->dyd.actvar.n = 2, but 'nactvar' = 1 still
     -- After 'j' is parsed, LexState->dyd.actvar.n = 3, but 'nactvar' = 1 still
     -- At the end of the statement 'nactvar' is set to 3.
     return i+j
   end
   -- As before the leaveblock() calls removevars() which resets
   -- LexState->dyd.actvar.n to 0 (the value before testfunc() was parsed)

   A rough debug trace of the above gives:
   function testfunc()
     -- open_func -> fs->firstlocal set to 0 (ls->dyd->actvar.n), and
fs->nactvar reset to 0
     local function tryme()
       -- new_localvar -> registering var tryme fs->f->locvars[0] at
ls->dyd->actvar.arr[0]
       -- new_localvar -> ls->dyd->actvar.n set to 1
       -- adjustlocalvars -> set fs->nactvar to 1
       -- open_func -> fs->firstlocal set to 1 (ls->dyd->actvar.n),
and fs->nactvar reset to 0
       -- adjustlocalvars -> set fs->nactvar to 0 (no parameters)
       local i,j = 5,6
       -- new_localvar -> registering var i fs->f->locvars[0] at
ls->dyd->actvar.arr[1]
       -- new_localvar -> ls->dyd->actvar.n set to 2
       -- new_localvar -> registering var j fs->f->locvars[1] at
ls->dyd->actvar.arr[2]
       -- new_localvar -> ls->dyd->actvar.n set to 3
       -- adjustlocalvars -> set fs->nactvar to 2
       return i,j
       -- removevars -> reset fs->nactvar to 0
     end
     local i, j = tryme()
     -- new_localvar -> registering var i fs->f->locvars[1] at
ls->dyd->actvar.arr[1]
     -- new_localvar -> ls->dyd->actvar.n set to 2
     -- new_localvar -> registering var j fs->f->locvars[2] at
ls->dyd->actvar.arr[2]
     -- new_localvar -> ls->dyd->actvar.n set to 3
     -- adjustlocalvars -> set fs->nactvar to 3
     return i+j
     -- removevars -> reset fs->nactvar to 0
   end
 */

Thanks and Regards
Dibyendu