A note on automatic resource management

[Florian Weimer <fw@...>]

Quite deliberately, Lua has no means to write finalizers in Lua.
User-defined finalization is an inherently concurrent activity, and it
is impossible to provide it in a safe way without basic concurrency
facilities (such as mutexes or more elaborate concurrent data
structures).

Deterministic finalization, using destructors or something like the C#
"using" construct, seems straightforward at first.  Various approaches
to syntax have been discussed on this list in the past.  However,
coroutines make things significantly more complicated because the
coroutine yield and resume provide a way to exit and enter scopes that
do not fit the simple construct/destruct model which is inherent to
the "using" construct.  Clearly, the destructor should not be called
on yield because a subsequent resume might still need the resource.

This is why Scheme's dynamic-wind has before and after clauses, unlike
Lua's xpcall.  However, this only resolves the issue under the
assumption that the Scheme image is the whole world, and no changes
outside of it are relevant.  For instance, let's assume that we have a
coroutine which processes a file.  When the coroutine is resumed after
yielding, the file needs to be closed and reopened.  However, in the
mean time, it may have vanished.  Similarly, it depends on the purpose
of a lock if it should be released and acquired when the executions
moves out of and into the coroutine.

One way to deal with this is to run the stack-based destructors when a
coroutine is finalized.  However, this brings us back to the initial
observation: this would allow programmers to write finalizers in Lua,
and as we've seen, there is currently no safe way to do that.

C# does not suffer from this problem because threads and their stacks
are part of the root set as long as they are running.  Lua coroutines
are subject to garbage collection when they are not active.  If the
stack is part of the root set, the question of whether to unwind it as
part of garbage collection does not arise.  (This assumes that the CLR
does not support thread cancellation, which it probably doesn't
because it's difficult to get the semantics right.)  But objects
holding native resources still need finalizers because the debugger
could be used to break invariants.  And object references do not have
to be restricted to the stack anyway.

So unless Lua gains the required facilities for (non-preemptive)
concurrency, there seems to be no good way to add automatic resource
management to the language.  I set of workarounds is possible and
achieves almost the same thing, assuming that exceptions are rare and
heaps are small: Objects which act as proxies for natives resources
should provide both an explicit "close" operation (as a replacement
for the destructor), and a native finalizer.  Programmers need to
manually invoke the close operation on regular scope exit.  Exception
handlers should perform a full garbage collection (I think Roberto
suggested this before), then there is no need to install exception
handlers to clean up the native resources locally.  Of course, this
only works if the program does not trigger too many exceptions,
otherwise most of the execution time will be spend in garbage
collection.  Fortunately, most Lua programs have smallish heaps, so
the overhead of doing this should be bearable.

(I thought I'd post this here instead on my dysfunctional blog.  Hope
you liked it.)