Re: Coroutines [was: snapshot of Lua 4.1 (work)]

[Jean-Claude Wippler <jcw@...>]

John Belmonte <jvb@prairienet.org> wrote:

[coro's]
>This issue has already been hashed out.

Thanks for the info - I had not seen this discussion before.

>But here was Roberto's final word on it:
>
>> > Presumably the hard bit is making it stay stackless when it calls C;
>>
>> The problem is that there are lots of places where Lua can make a
>> call to C. Any tagmethod can be a C function, and so most basic operations
>> may call C. Currently we handle calls to C and to Lua in the same way.
>> To change that would break a nice "simetry" in the code, and would need
>> lots of changes in the code. (It could also compromise performance...)

Calls to C are not a problem.  Calls from C to Lua are.  How would the
symmetry in the code be affected?

>> We had three goals in mind: restricted co-rotines (only Lua), generic
>> co-routines, and multi-thread. Lua stackless would demand lots of work,
>> and  would solve only the first goal; it would be useless for the other
>> two.  Multiple stacks solves the other two goals (with a small external
>> help), it is quite simple to implement, and has no impact in performance.

The first goal is actually a major one: one way to use scripting, is to
code only small (i.e. atomic) pieces in C.  So C is a way to add
"primitives" to the scripting language (the other main use is to add
bindings to existing libs).  With that perspective, co-routines are not
needed in C: if some C logic requires the ability to suspend/resume, one
can try to recode the logic as two C functions and make the state
explicit (or go for multi-threading).  To put it differently: one could
achieve excellent parallelism with just Lua-level coroutines (async I/O +
"restricted" coro's would be an awesome combo, IMO).

Goal two comes at a considerable cost: portability, or rather lack of it.
 C-level stack switching requires machine code, and introduces the tricky
issue of preventing overflow in all stacks.  It would only be needed when
C code calls back into Lua which *then* suspends, which is not too likely
when bringing in existing C libraries, nor when C code consists of small
bits of functionality (e.g. speed-ups).  It should be possible to detect
and prevent the use of suspend / resume in such a Lua -> C -> Lua nested
case (just like tag methods prevent runaway recursion).

Goal three is not really related to the above: Lua uses no globals, so
threading in itself already comes for free.  The lock/unlock hooks make
it possible to share data between multiple threads.  This issue does not
require stack switching, and is in essence already solved in the 4.1-work
preview.

While complete stacklessness ("continuations") is no doubt very hard to
achieve, I'm not convinced that one could not chase down all luaD_call
spots and resolve the issues for the restricted coro case.  Tag methods
must be dealt with, but many appear to be tail calls.  These could be
made stackless by pushing the call onto the VM execution stack and
*returning* to the VM, instead of calling into one more level of C.

-jcw