RE: Reducing memory consumption

["Nick Trout" <nick@...>]

> So, this is quite an opened question and I have few ideas but I'd like
> to throw it out open... What ideas do people have for reducing the
> memory consumption of Lua in working scripts?

I've just been thinking about scripting and binding optimisations. When
you register a C function Lua copies the function name you gave it so
you have 2 copies of string in memory. In constrained environments, with
a couple of thousand registered functions, along with the closures this
can add up. So I was thinking of trying to remove the necessity for
strings at all and also remove or reduce closures.

If I have a list of functions to register in C this amounts to something
_like_:

// C
Lua_function_info fns[] = {
	{ "ThisFunctionNameIsVeryDescriptive1", myFnCallback1 },
	{ "ThisFunctionNameIsVeryDescriptive2", myFnCallback2 },
	{ "ThisFunctionNameIsVeryDescriptive3", myFnCallback3 },
	{ "ThisFunctionNameIsVeryDescriptive4", myFnCallback4 },
	{ "ThisFunctionNameIsVeryDescriptiveN", myFnCallbackN },
	{ 0, 0 }
};

lua_register_list(fns);

So we end up with duplicate copies of all those strings (because Lua
copies the function name) and closures around the C functions (32 bytes
each). I can't register these as local variables (so they're indexed) so
I can't get the space back. (Perhaps I could somehow register them
through the debug library?)

Perhaps Lua should have support so that you can register C functions
into a lua_State as local variables rather than globals?

Alternatively, you could, once the script has started, include something
like:

-- Lua
local ThisFunctionNameXXX1 = ThisFunctionNameIsVeryDescriptive1
ThisFunctionNameIsVeryDescriptive1 = nil
local ThisFunctionNameXXX2 = ThisFunctionNameIsVeryDescriptive2
ThisFunctionNameIsVeryDescriptive2 = nil
-- Etc.

And this would get you the Lua space back for the global variable
strings after the first garbage collection. Also function invocations
would be faster because we're avoiding a global lookup. But the C
registration strings and closures are still hanging around.

One solution which harks back to the days of BASIC programming is to
"compress" your script. So we could go through and on the C and script
side reduce name lengths, e.g.

	ThisFunctionNameIsVeryDescriptive1 -> fn1
	ThisFunctionNameIsVeryDescriptive2 -> fn2
	ThisFunctionNameIsVeryDescriptiveN -> fnN

This would save space on both sides, but is not very practical for
development, but could possibly be done as a pre-process (Lua side)
before the scripts are compiled. Perhaps macros (C side) could be used
to have 2 different version, one for development and one for release.
But we still end up with lots of closures here.

So the ideal solution would be no closures and no registered strings;
ahhh a static language! Or perhaps this is just a new type of closure.
If we don't want upvalues, the only purpose of our closure is to pass
some arguments to a C function. Perhaps if we enumerate all of our C
side functions we could substitute Lua invocations for the following
(during a pre-process):

-- Lua
function foo()
	local bar = 3
	ThisFunctionNameIsVeryDescriptive1(77, bar, "string")
end

The above, pre-processed, becomes:

-- Lua
local ThisFunctionNameIsVeryDescriptive1 = 1
local ThisFunctionNameIsVeryDescriptive2 = 2
local ThisFunctionNameIsVeryDescriptiveN = N

function foo()
	local bar = 3
	__dispatch(77, bar, "string",
ThisFunctionNameIsVeryDescriptive1)
end

Then we have a C function registered which pops off the first argument
from the stack (the last arg, ThisFunctionNameIsVeryDescriptive1 = 1).
Pulls out our Lua callback function (which has the same declaration as a
lua_registered function) and calls it, e.g.

// C
int dispatch(lua_State *L)
{
	int fnenum = lua_tonumber(L, -1); // I think this pops?

	int (*fn_to_call)(lua_State*) = my_fn_list[fnenum];

	fn_to_call(L);
}

So now we can do away with all the function registration, we just need a
plain array of all the C functions. But the only bind is that all of the
C function names have to be exported and then the Lua function calls
must be substituted in the scripts. But, we save all the string space
and there no closures.

Sorry, I'm just thinking aloud at the moment. Is there an easier way to
do this?

Nick