GC causes unexpected reentrancy

[Mark Hamburg <mhamburg@...>]

Some background: I'm working on a multi-threaded (as in system threads)
application in which multiple threads can all access the same Lua universe
through multiple lua_States. They end up serializing at the Lua API because
of the way lua_lock works, but that's not important.

Now, consider the case where I've got userdata that needs to store Lua
references of some sort. In my case, it's that I've got reference counts on
userdata items for non-Lua references and any items with non-zero reference
counts have to be entered into a Lua table so that they don't get garbage
collected. I believe, however, that the same problems would arise if the
userdata items held Lua refs.

Because the application is multi-threaded, when I update the Lua data
structures (userdata retention table or a ref table), I need to bracket
those operations with mutex locks.

So, now consider the following path....

* Retain an object (or release it)
    * Locks the retention table mutex
        * Calls to Lua to modify the table
            * Lua invokes the GC
                * The GC invokes a userdata GC method
                    * Userdata GC method releases an object
                        * Attempt to lock the retention table mutex

Deadlock.

For my particular usage, I might be able to resolve my problem by using a
recursive mutex for the retention table, but that gets a bit delicate. For
the ref case, I've looked at luaL_ref and it makes multiple Lua API calls
and it doesn't seem wise to me to re-enter any of that code so a recursive
mutex wouldn't help.

Any suggestions?

My current thought is that I should look at the sweep phase of the garbage
collector and work on making the portion that calls the GC methods for
userdata run in a separate thread. That, however, is going to require
changing the Lua sources.

Mark