state of the Lua nation on resource cleanup

[John Belmonte <john@...>]

With the Gems article on exceptions published-- a large portion of it
on the topic of cleanup in the face of exceptions-- I spent some time
over the holidays catching up with various writings and
implementations appearing in the past year related to resource
finalization.  There was a try/catch/finally implementation by Hu
Qiwei, the "finalization stack" API by Nodir Temirhodzhaev, and
finalize/guard blocks introduced by Alex Mania.  While I feel strongly
that Lua needs some kind of improvement in this area, none of these
appear quite in line with the design philosophies of the language.
David Manura and I had a bit of discussion off list on possible
constructs and syntax, but I noticed we weren't converging on a
solution, and I expect such an effort would only become more difficult
with a wider audience.  So taking a step back I decided to frame this
as sequence of design decisions, starting with the essential question
of "do we need this?" and ending with less important details that are
going to be up to the personal preference of the Lua authors anyway :)

    <http://john.neggie.net/2009/lua/resource_finalization>   (also
included in plain text below)

Besides opinions from the community, I'm hoping for some word from the
Lua authors like one of "working on a solution", "Lua doesn't need
improvement here", "see the need for it but don't have a design in
mind", etc.

Regards,
--John


=== Lua: Improving deterministic resource cleanup ===

== Design decisions ==

Ordered roughly by importance, here is the sequence of design
decisions for a new resource cleanup (or finalization) mechanism in
Lua.  If the Lua authors and finalization "usual suspects" in the
community could agree on the answers to at least the first four items
perhaps we could proceed in making this improvement to Lua.

   1. Should the Lua language provide better support for deterministic
resource cleanup?

      Yes.  Many applications expose scarce resources to the scripting
environment such as locks; handles for files, memory, and database
transactions; etc.  Such resources need to be released promptly and
deterministically after use, and despite any exceptional condition in
the program.  Of course this can be accomplished with Lua as it is,
but the resulting code is hard to follow [1], highly error prone, and
makes heavy use of protected calls-- limiting potential use of
coroutines.

   2. Should this enhancement allow distinguishing of "exit" from
"exit by error"?

      Yes.  A common use of the popular "try-catch" construct is
simply to perform a side effect in the case of an error (for example,
to roll back a transaction).  The error doesn't actually need to be
"caught" in the sense that we don't need to transform it, suppress it,
or even know any details about it.  Handling these situations with a
unique cleanup idiom makes the program more clear and reduces
programming errors, such as neglecting to re-raise an exception.

   3. Should the error object be provided to the cleanup code?

      Probably.  This seems harmless and would open the door for
transformation or suppression of errors.  Normally this is
accomplished with a try-catch construct, which Lua doesn't have.  If
such a construct were separately added, indeed there is no reason to
make the error object available in the cleanup case.

   4. Should cleanup code be injected by declarative syntax or meta mechanism?

      This being Lua, the answer is meta mechanism.  By way of
contrast, the experimental finalize/guard enhancement presented on the
Lua mailing list is an example of declarative syntax.  Cleanup code is
added in special blocks opened with new "finalize" or "guard"
keywords:

      function foo()
        local lock = aquire_lock()
        finalize lock:release() end
        ...
      end

      The issue with a declarative cleanup block is that the resource
user is compelled to write code.  Another example of declarative style
is the popular try-catch-finally construct-- argued as poorly suited
for resource finalization by the authors of the D programming
language, "Exceptions in Lua" Gems article, and Lua finalize/guard
patch.  Better would be a mechanism where the scope was made aware of
certain objects, notifying them on exit-- in other words a
user-defined hook for scope exits.  If such a mechanism existed, a
declarative cleanup style could easily be implemented on top of it.
An example of this is the scope manager pattern from the "Exceptions
in Lua" Gems article.

   5. Should the scoped object be designated by...

         1. a "scoped" variable class?
            That is, a keyword "scoped" which acts exactly as "local",
except that the value is noted for signaling on scope exit or
(perhaps, with additional implementation difficulty) when the variable
is reassigned.  If there are several of these within a block level,
they should probably be considered as sequentially nested.  That is,
if we have "scoped a; scoped b", b would be cleaned up first, and any
error in the cleanup code would be propagated to a's cleanup.

         2. metamethod magic?
            Similar to above but doesn't require a new keyword.  For
example, add a new metamethod called "__exit".  Upon assignment of a
local variable, the value is checked for this metamethod.  If it
exists, the value is noted for signaling on exit.  The disadvantage
here is that neither the compiler nor person reading the code can know
they are dealing with scoped objects.  Also seems intrusive to a
critical code path, namely locals assignment.

         3. a "with" block?
            This would be a new type of do-end block which essentially
means "with a certain resource, do something, and then free the
resource".  Python has this, and Metalua offers it as a language
extension.

                i. should it take the form "with VARLIST = EXP" or
"with EXP [as VARLIST]"?
                  The latter naturally emphasizes the importance of
the scoped object, treating the name as secondary and optional.

      It should be noted there exist many more variants on the above
three themes.  Overall, the "with" block seems preferable since it is
explicit and provides a syntax which emphasizes the scoped value over
any variable which it happens to be assigned to-- and in fact allows
eliding of the variable altogether, which is useful in many common
cases such as holding a lock.

   6. Should scope exit be signaled by callable interface, fixed
method name, or new metamethod?

      This applies to "scoped" var and "with" block solutions--
various trade-offs here.  Callable allows simple function variables
and lambdas to be provides as the scoped EXP.  However this prevents
potential uses of the callable interface within the block.  A fixed
method name (Metalua's approach) removes this restriction while
avoiding the need to deal with metatables.  Using a new metamethod,
say __exit, is the most robust solution but can be tedious in simple
cases.

   7. Should there be a scope entrance hook?
      No. This can be effected by requiring a call to instantiate the
resource or manager, and following the convention of only making this
call from the scoped EXP.

----
   [1] Challenge: implement Alex Mania's "appendud" example function
in plain Lua while keeping the code easy to follow.  See
<http://lua-users.org/lists/lua-l/2008-02/msg00243.html>.