"propagateall" enter infinite loop during gc atomic phase in a certain situation

[张伟智 <robotkzhang@...>]

Lua version is 5.2.4
During gc atomic phase, markbeingfnz() mark mark all objects in list of tobefnz:
311 static void markbeingfnz (global_State *g) {
312   GCObject *o;
313   for (o = g->tobefnz; o != NULL; o = gch(o)->next) {
314     makewhite(g, o);
315     reallymarkobject(g, o);
316   }
317 }
Suppose "Mt" is the metatable of userdata A and B,
and "Mt" also has __gc finalizer, "Mt" will be linked to g->gray twice,
first by userdata B   : reallymarkobject(g, B)   -> markobject(g, B->metatable) -> linktable
second by "Mt" itself: reallymarkobject(g, Mt) -> linktable

////// lbaselib.c
+static int luaB_myudata (lua_State *L) {
+       lua_newuserdata(L, 3);
+       lua_pushvalue(L, 1); //mt
+       lua_setmetatable(L, -2);
+       return 1;
+}

static const luaL_Reg base_funcs[] = {
+  {"myudata", luaB_myudata},


////// test.lua
 1 local MtOfMt = {
 2         __gc = function(Obj)  end
 3 }
 4
 5 local Mt = setmetatable({
 6         __gc = function(Obj) end
 7 }, MtOfMt)
 8
 9 local A = myudata(Mt);
10 local B = myudata(Mt);
11 A = nil; B = nil; Mt = nil;
12 collectgarbage("collect")

tobefnz --> B -> A -> Mt
After marking B, gray list is :
gray -> Mt

when mark Mt, Mt will be link to gray again,
gray -> Mt---
            ↑___|
Mt's gclist point to itself.
There is a loop in gray list, propagateall will enter infinite loop.

566 static void propagateall (global_State *g) {
567   while (g->gray) propagatemark(g);
568 }