Re: GC problem with lot of allocated memory

[Sean Conner <sean@...>]

It was thus said that the Great Rodrigo Azevedo once stated:
> The code
> 
> N = 1.0e7
> C = {}
> for i=1,N do C[i] = i end
> print(collectgarbage("count"))
> for i=1,100*N do
>         local a = {1,2,3,4}
> end
> print(collectgarbage("count"))
> 
> prints
> 
> 262166.11035156
> 445760.78710938
> 
> problem: when Lua uses lot of memory, the creation of temporary objects can
> consume all your memory. I think CG should collect the objects fast enough,
> but it does not. A perfectly OK program can't run because of the amount of
> garbage to be collected consumes all memory.

  The code presented does not generate any garbage so there's not much to
collect.  To do some tests on my system, I had to change the code a bit:

N = 25000   
C = {}
for i=1,N do C[i] = i end
print(collectgarbage("count"))
for i=1,100*N do
        local a = {1,2,3,4}
        a = nil
        if i % 1000 == 0 then 
	   collectgarbage('count')
          --collectgarbage('step')
        end
end
print(collectgarbage("count"))

This let it run without killing my system and still give similar (if
smaller) results.  I then ran (with collectgarbage('count') uncommented) it:

[spc]lucy:/tmp>time lua /tmp/mem2.lua
403.1630859375
742.5458984375

real    0m1.600s
user    0m1.597s
sys     0m0.002s

Then then switched to calling collectgarbage('step'):

[spc]lucy:/tmp>time lua /tmp/mem2.lua
403.1953125
473.984375

real    0m1.749s
user    0m1.749s
sys     0m0.001s

A tad slower, but the memory usage did drop.  I then replaced 'step' with
'collect' just to be thorough:

[spc]lucy:/tmp>time lua mem2.lua
403.1884765625
401.7900390625

real    0m1.800s
user    0m1.798s
sys     0m0.001s

There are two values you can use to tweak the GC in Lua and it takes a bit
of finessing to fine tune it.  

> question: There is some place of Lua source code that I can modify to keep
> the memory usage as close as possible to 262166.11035156 without doing my
> program 3x slowly?

  Tweak the setpause and setstepmul values?  Use 'step' instead of
'collect'?  This will take some experimenting to know for sure.

  -spc