Re: Interesting interaction between debug hooks, the garbage collector, and "open" VM instructions

[Peter Cawley <lua@...>]

The modified opcodes change the 'A' field of the vararg instruction
from the top of the stack to the bottom, meaning that after the vararg
instruction is executed, L->top ends up smaller than ci->top, but
there are still important values between L->top and ci->top. This
cannot happen with code generated by Lua's code generator as either it
will always put vararg instructions at the top of the stack, or it
will put them at such a point on the stack such that anything between
L->top and ci->top is not important.

One obvious question is why you would want important values between
L->top and ci->top after a vararg instruction, as surely they were
unimportant as they could have been overwritten by the vararg
instruction? I can think of two situations which make use of important
values between L->top and ci->top:
1) An interesting implementation of function default arguments. The
current paradigm is "arg = arg or default", but this has issues when
arg is false or nil. An alternative is to change the function from
having explicit arguments to having variable arguments, load the
default values into stack slots 0 through N, and then use a vararg
instruction at the bottom of the stack to overwrite some of the
default values and leave the other default values intact (obviously
the vararg instruction would have to be followed by an open
instruction to correct L->top, and a setlist instruction with the 'A'
field pointing to a non-table would do the trick).
2) An implementation of select('#', ...) in Lua rather than in C.
Using a variation of technique 1, a Lua function could determine if it
was called with zero arguments or more than zero arguments (even if
the arguments are nils). Combined with a recursive call, this could
count the number of arguments with correct behaviour for trailing
nils.

The debug hook then comes along with L->top less than ci->top, and
does a GC cycle. When the GC is marking a thread, it traverses the
stack of the thread, marking everything up to L->top, and then if
ci->top is higher than L->top, it nils everything between L->top and
ci->top. This of course sets to nil the important values which were on
the stack between L->top and ci->top. Without the debug hook, the
important values between L->top and ci->top are preserved.

It is arguable whether or not it is valid to have important values
above L->top after a vararg instruction, but I believe that it does
allow for a few interesting things.

On another note, Patrick's patch fails to take into account situations
like this:

function Example(...)
  print(...)
  local a, b, c, d, e, f, g
end
Example("hello", "world")

The above code should print "hello world", but with Patrick's patch,
it prints "hello world nil nil nil nil".


On Sun, Feb 8, 2009 at 9:24 AM, Patrick Donnelly
<batrick.donnelly@gmail.com> wrote:
> On Sat, Feb 7, 2009 at 5:11 AM, Roberto Ierusalimschy
> <roberto@inf.puc-rio.br> wrote:
>>> In extremely convoluted situations, if a debug hook is called
>>> immediately before an open VM instruction, and then proceeds to do a
>>> garbage collection, then the behaviour of the program can change. This
>>> is shown in the example code below, which will print "fail" if the
>>> debug hook is set, and "pass" if it isn't:
>>>
>>> [...]
>>>
>>>   // Change a the {...} construct to use the un-used b through to f stack slots.
>>> [...]
>>
>> Can you explain the problem? (I mean, why your example has that
>> behavior.)  May this problem happen with "original" Lua opcodes?
>>
>> -- Roberto
>>
>
> This is caused by the top of the stack being reduced below the top
> slot for the running function. The string and table will both be
> collected (during the hook) by the GC before the SETLIST operation is
> executed. What follows is a patch which corrects the problem.
>
> batrick@waterdeep:/src/lua-5.1.4/src$ diff -Naur /home/batrick/tmp/lvm.c~ lvm.c
> --- /home/batrick/tmp/lvm.c~    2009-02-08 01:35:04.000000000 -0700
> +++ lvm.c       2009-02-08 01:35:04.000000000 -0700
> @@ -745,7 +745,8 @@
>           Protect(luaD_checkstack(L, n));
>           ra = RA(i);  /* previous call may change the stack */
>           b = n;
> -          L->top = ra + n;
> +          if (L->top < ra+n)
> +            L->top = ra + n;
>         }
>         for (j = 0; j < b; j++) {
>           if (j < n) {
>
> I don't believe there is any way to corrupt Lua using this.
>
> --
> -Patrick Donnelly
>
> "One of the lessons of history is that nothing is often a good thing
> to do and always a clever thing to say."
>
> -Will Durant
>