When memory usage exceeds 2G, each call to lua_gc(LUA_GCSTEP)
becomes a full gc cycle (retval = 1).
#define setthreshold(g) (g->GCthreshold = (g->estimate/100) * g->gcpause)
Are you on a 32-bit or 64-bit platform ?
On a 32-bit platform, it is impossible or unreasonable to use more
that 2 GiB of memory in a application.
On a 64-bit platform, it should already work according to my analysis.
Both GCthreshold and estimate fields of struct global_State are of
type lu_mem == LUAI_UMEM, which should be defined as size_t (64 bit
integer) on a 64-bit platform. So the computation should be performed
in 64 bits and no overflow possible.
