Inside the __gc
function for an object (whether table or userdata), getmetatable(self)
has the same value it had just before the object was collected, and
getmetatable(self).__gc is the __gc method being executed. Nor is the
metatable changed after the __gc method completes.
Nothing forbids the getmetatable(self) to return the metatable of the object it had before it was collected: self is referencing the closure in which the finalizer was created and is independant of the object itself (a finalizer function may be created and set in a (meta)table long before the object is created and the m(eta)table is associated with the object by using setmetatable(object, (meta)table);
Self is not the same as the object (o) passed in the 1st parameter of the finalizer, whose metatable may still have been cleared.
As well nothing forbids getmetatable(o) to return the effective metatable of the object (o) before it was modified by the GC: the GC prepares an environment for calling the finalizer, in which the getmetatable function will be found that will still be able to return that value that the GC has kept.
But in my opinion all this is unnecessarily complicate. It would be much simpler if the finalizer indicated to the GC that the object must not be swept, by just returning a non-nil value. If the finalizer does not thing, or reaches the end of its code without using a return instruction, or if it uses "return nil", the effect is the same: the first upvalue returned will be nil and the object must then be swept.
If the finalizer just "return true", it clearly indicates that the object must be kept; the GC does not have to create a specific environment, the finalizer does not have to inspect the object state on return, does not have to track the usage of setmetatable() by the finalizer.
The GC will take its decision to sweep or keep the object only by looking at the first upvalue returned by the finalizer. This is much clearer! And the GC does not even have to change any internal state of the object before calling the finalizer, so this is also more efficient.
Finalizers could also return other interesting status for the object to keep (e.g. indicating not just the fact that it must be kept, but also, for example returning if it should be kept in the active generation or placed in an older generation (to be finalized later, but much less urgently: e.g. if it returns false it is kept in the current generation, if it returns true, it is kept in the older generation, for Lua implementations that support generations in their GC; the non-nil return value is then a hint given to the GC about what to do with the preserved object, because that object will be finalized again and again, at every GC-cycle, if the finalizer constantly returns a non-nil value when it is called ! The hint can be used to reduce the frequency of calls to this finalizer, which is acting like a coroutine running most often at unpredictable times, but indefinitely without ever really terminating as long as it returns a non-nil value or does any other action indicating to the GC that the dead object must be kept).
