Re: Feature request: more integration of Lua and C memory management

[Jamie Webb <j@...>]

On Thursday 10 June 2004 03:55, Bilyk, Alex wrote:
> This is somewhat not clear to me. We know how Lua deals with its
> references. In order to solve your issues we have to also impose certain
> protocol on the application with respect to GC and reference trecking
> strategy on the host side. Right? Without such protocol/requirement how can
> one guaranty that host application references are in some sort of synch
> with Lua side. For instance in my app I also use objects that are made of
> pairs <userdata(UD), table(T)>. One protocol I use is this
>
> 1. UD is referenced only once on the host side unless referenced by another
> object in my object universe. 2. UD's GC is triggered by T's GC. T life
> time implies UD life time, while the other way around is not true. 3. UD
> knows its T and vise versa.
>
> The GC for this scheme works well -- cycles or no cycles (because of item
> #1 and #2). That is, I can reference any Lua objects from any of my <UD, T>
> objects and everything would get collected in its due time following
> standard Lua GC protocol. Basically, whatever T holds in it, can also be
> said as being held by UD (it's fully accessible from UD) . Whenever T is
> collected with everything it held inside, UD is collected and that is the
> end of it. This is an application specific protocol that may not suit
> everyone. I use another protocol for other kinds of objects -- it's also
> application specific. What scheme do you have in mind that would satisfy
> everyone? IOW, what protocol with regard to userdata would one have to
> follow on C side to have everything working the way you envision?

Agreed, your scheme is correct with respect to GC, but it comes under my 
definition of 'not easy':

- The references held by the userdata are not protected from Lua scripts 
unless you add an extra proxy table. At that point you would have 4 
allocations per userdata: the userdata itself, the peer table, the proxy 
table, the proxy metatable. That's somewhat heavy.

- Method calls involve 3 table lookups (table, metatable, table of methods) to 
get to the method. Then another metatable access to get the peer if using a 
proxy, then you have to index your peer in order to recover the userdata.

- Suppose you're following a trail of pointers on the C side (e.g. object 
method calls). To write to any of those objects, you either have to follow 
the same trail with the Lua stack (slow, impractical), or presumably you have 
a fully weak table in the registry mapping pointers back to peers (still more 
work on object creation), the use of which costs a further 3 table accesses 
(registry, weak table, peer).

In short, it's even slower and heavier than using unique metatables.

The protocol I'm suggesting:

- Userdatas have attached to them an 'array part'. The array should be 
accessible given only a pointer to the userdata (but on second thoughts, 
counting backwards was a stupid idea). This doesn't really reduce the safety 
any, since presumably the code is going to manipulate the raw part using that 
pointer anyway. Accesses to this array are as fast as to upvalues (and indeed 
Mark Hamburg originally described this scheme as 'userdata upvalues').

- Per-object references should be kept in the new array part of the userdata. 
When an object method is called by Lua, it must be live, so every object it 
references must be live too. Everything is fine as long as a pointer to a 
userdata is never stored without a corresponding reference in the array part.

- Any static references or references from pure C objects to userdatas should 
be obtained with luaL_ref. If the lifetime of a C object depends on a 
userdata, then it should not hold references, and instead the parent userdata 
should. Yes, that could mean breaking encapsulation, but its better that that 
be done in C rather than spilling into Lua. I don't think there's any 
alternative with any protocol as long as a memory management boundary exists.

- Recovering the Lua value of a userdata from its pointer becomes easy, since 
'self' could be stored in the array. Alternatively, the API could provide a 
lua_pushuserdata that does this by pointer arithmetic.

I think that would satisfy everyone because it is fast, light, correct, and 
easy to program, and I can't think of any other criteria that one might use 
to choose a protocol.

-- Jamie Webb