Re: PATCH: Fully Resumable VM (yield across pcall/callback/meta/iter)

[Mike Pall <mikelu-0502@...>]

Hi,

Javier Guerra wrote:
> this scares me a bit...  i was aware of the last problem (yielding across C 
> functions), but none of the others.  the first two are specially worrysome

Maybe it was not documented as such (yet). But the (old) error from yield
is pretty obvious: "attempt to yield across metamethod/C-call boundary"

The Wiki page has a long list of all combinations I found, where yielding
is possible or not. Without my patch that table is almost exclusively
filled with "No's".

Of course nobody will need *all* of these at once, but
1. it quickly gets annoying to work around every single case and
2. once the framework was in place it was pretty easy (read: 0-10 lines
   each) to make all metamethods and other stuff resumable.

My main intention was to fix the pcall/yield and the C-yield-and-resume
problems. Everything else is just a nice side-effect. :-)

> - for metamethods you mean any function in a metatable? or just the hooks?

In Lua 5 terminology "hooks" are debug hooks and have little in common
with metamethods. Metamethods are __index, __newindex, __add, __concat,
and so on.

Of course you can use tables instead of functions for __index and
__newindex (but not the others). No metamethod is called then. But this
is not sufficient for all cases (think about containers tranparently
mapping index operations to network calls for a remote database).

> on second thought, the methods used for OO-like constructs aren't in the 
> metatable, but in another table referenced by the __index hook; so i guess 
> we're safe here.

Yes, because these are not metamethods. But as soon as you use a function
(and not a table) for __index, any innocent expression such as t[x] may get
you into trouble.

Try this simple example with plain Lua 5.1 and then again with the patch
applied:

  local function get(t, x)
    coroutine.yield("GET:", t, x)
    return "foo"
  end

  local function cofunc()
    local t = setmetatable({}, {__index = get})
    print(t[1])
    return "end of coroutine"
  end

  local function coprint(co, ok, ...)
    if ok then print(...) else print("ERROR:", ...) end
    return ok and coroutine.status(co) ~= "dead"
  end

  local co = coroutine.create(cofunc)
  while coprint(co, coroutine.resume(co)) do end

[Dito for all other metamethods]

> - for "across iterator functions", do you mean "from an iterator"? or from 
> inside a for loop?  i use coroutine iterators, are those safe?

>From inside a for loop (not when you call it 'by hand'). Try this example:

  local function iter(t, x)
    local k, v = next(t, x)
    coroutine.yield("ITER:", t, x, "->", k, v)
    return k, v
  end

  local function cofunc()
    local t = {4, 5, 6}
    for k,v in iter,t,nil do
      print("t["..k.."] = ", v)
    end
    return "end of coroutine"
  end

  local function coprint(co, ok, ...)
    if ok then print(...) else print("ERROR:", ...) end
    return ok and coroutine.status(co) ~= "dead"
  end

  local co = coroutine.create(cofunc)
  while coprint(co, coroutine.resume(co)) do end

Yes, of course this is silly, since you can use next() directly here. But
iterators are a classic case for yielding (for line in sk:lines() do).

Or try this obfuscated one for fun: ;-)

  for k in coroutine.wrap(function()table.foreach(_G,coroutine.yield)end) do
    print(k)
  end

In fact I have a complete set of test cases that I need to polish a bit
before uploading.

Bye,
     Mike