Feature request: more integration of Lua and C memory management

[Jamie Webb <j@...>]

Many libraries need to hold references to Lua objects, and to expose userdatas 
to Lua code, and this creates a nightmare for memory management. Every object 
must be either under the control of the Lua GC, or managed by the C code. 
Consider some struct, which is heavily used and stored in both C and Lua 
code, and passed between the two often. There are two options:

- Manage the struct from C.
AFAICT, the only way to be sure that both Lua and C no longer contain 
references to an instance is to maintain a reference count (even if that's 
just a pair of flags indicating whether the object has Lua and C references). 
This appears to be necessary regardless of the memory management strategy 
used on the C side. There is then a possibility of an uncollectable reference 
cycle between C and Lua, which would be very expensive to detect. Requiring 
Lua users to avoid reference cycles would be onerous, bad practice, and 
potentially slow. I am particularly concered about reference cycles because I 
think that when closures are being passed to C code, they are almost 
inevitable, and they could potentially involve a large number of objects.

- Create the struct as a userdata.
Now everything is fine from the Lua side, but every time a reference is kept 
on the C side, it must be stored in a table that will be GCed along with the 
reference holder (which must itself be a userdata). That means having a 
unique metatable for each userdata (I don't think any scheme with weak tables 
will work correctly, and it would be too slow anyway). This though makes 
object creation too expensive (I'm expecting lots of little short-lived 
objects, and making them userdatas means I can't pool them). Also, this may 
require additional bookkeeping to link pointers back to userdatas. I don't 
believe any other solution (e.g. luaL_ref) can collect reference cycles. 

The only solution that I can see is to improve how userdatas hold references 
to other Lua objects. Probably the neatest way is to associate with each 
userdata an array (length specified when the userdata is created) of 
TObjects, in which it may store Lua objects, and which the garbage collector 
knows about. That array should be accessible given only a pointer to the 
userdata, and not the actual Lua object (e.g. by storing the array 
immediately before the userdata and counting backwards).

What I'd really like to see though is more options, to make it possible to 
find solutions to these sorts of problems that are both fast and correct. At 
the moment, Lua memory management seems very much 'off limits'. Examples 
include allowing a userdata to be returned to a pool rather than free'd when 
it is GCed, and allowing C objects to take part in marking. C code shouldn't 
be reduced to the efficiency of a scripting language just because it 
exchanges objects with one.

Some of the isues I've mentioned here have been described before by e.g. Rici 
Lake and Mark Hamburg, but there doesn't appear to have been any resolution, 
or any input from the Lua authors.

-- Jamie Webb