Re: Research on Lua programs with excessive memory consumption

[Matthew Wild <mwild1@...>]

Hi,

On 2 May 2014 20:22, Pablo Musa <pmusa@inf.puc-rio.br> wrote:
> Hi everybody,
> I am doing some research about memory problems in Lua, more specifically on
> memory waste due to objects that will never be used again. My idea is to
> create a library to help "memory leak"* detection in Lua.

See:

  - http://code.matthewwild.co.uk/luatraverse/
  - http://code.matthewwild.co.uk/lua-getsize/

> In this research I would like to first analyze some real problems and then
> propose a solution.
> If you have a program which is using more memory than it should, can you
> please send it to me with as many details as possible?!!

Occasionally we have people with memory usage issues in Prosody. For
debugging these we have a small script based on the above projects
which will dump the Lua state as a graph to a file, with each entry
containing data such as the object type, size, etc. We then analyze
these using scripts and even a GUI tool to determine which tables are
accounting for the most memory usage, etc.

I don't have a sample application to give you, but I have a real-world
story of a "memory leak" we had once in Prosody. LuaExpat creates
"parser" objects, which call Lua handler functions when various events
occur. It stores these using luaL_ref(). Because of this there is
always a strong reference to the handler functions, as far as the GC
is concerned. The parser objects do remove this reference in their
__gc, however if the handler functions have the parser as an upvalue
(or, as in our case, hold a reference to it indirectly through another
table or function) then __gc won't get called. Then we have a cyclical
reference that can never be cleaned up by the GC. The result was a
slow leak of parser objects and handler functions.

> * I do not like the term "memory leak" for this problem, but that is not the
> focus of this email.

I believe Roberto likes to call it "hoarding":
http://lua-users.org/lists/lua-l/2012-03/msg00482.html :)

Regards,
Matthew