Garbage collector behaviour changed from 5.1 to 5.2

[Robert Norris <rob@...>]

Hi all,

Pioneer[1] is a game written in C++ that uses Lua for its scripting.
We expose game objects to the Lua environment using "boxed pointers".
We upgraded Lua from 5.1.4 to 5.2.0 a couple of weeks ago and have
noticed substantial increases in memory usage since them, to the point
that one of our developers on a machine with not much memory now can't
play because the game causes his machine to swap to death.

Investigating has uncovered a few potential problems with the way our
Lua bindings are written which we intend to investigate - mostly that
we allocate more game objects than we probably should and we certainly
need to limit the amount of memory Lua can use. There is however a
definite change in the way the garbage collector behaves, which can be
demonstrated by the short program available here:

  http://pastebin.com/UAgtS3PX

As you can see, while the 5.1 collector would regularly clean up and
keep memory usage fairly flat, the 5.2 collector rarely runs and when
it does it does not clean up much. I have tried to change the pause
and step to mimic the old behaviour but have not found a way that
helps. I also tried a custom allocator that caps memory usage but
since the emergency collector will not clean up objects with
finalisers it does not help much.

I would like to hear any ideas anyone my have on how to fix this.
Tuning the collector would be the nicest thing in the short term, but
I'm also prepared to accept that we may be fundamentally doing
something wrong in the way we expose our objects to Lua. All ideas
gratefully received :)

Cheers,
Rob.

1. http://pioneerspacesim.net/