Re: [Experiment] Module system based on static hashing

[Sean Conner <sean@...>]

It was thus said that the Great Stefan once stated:
> Hello,
> 
> I've conducted an experiment to speed up Lua state creation with static
> hash tables generated by GNU gperf and got some interesting results.
> 
> Introduction
> ============
> 
> Lua states offer a very light-weight way to execute independent scripts,
> which is a much desirable feature for programs that execute a large
> number of them (e.g. web servers).
> 
> Unfortunately the standard library is too small for many tasks and
> adding modules by hand is quite a hassle. Furthermore are dynamic
> libraries a platform-dependent mess.
> 
> The goal of this experiment was to find a way to add many more functions
> to Lua a) without using dynamic loading and b) without slowing down the
> creation of new states.
> 
> luaL_openlibs loads all functions, tables and values such as print,
> string, _VERSION, math.pi etc. that make up the standard library into
> the Lua state so that the script can access them via table lookups.
> 
> But rarely does a script use ALL of them and the more functions get
> added, the more unnecessary work luaL_openli has to do.
> So, the less unused Lua values get loaded into RAM, the better.
> 
> OK, so what if we don't actually load them and just set a metatable
> with a __index metamethod that fetches the values as the script needs
> them? The script won't notice absent values it doesn't use -- Great!

  I have a similar issue at work, although it's not speed reasons but
installation reasons (the less files to install, the better).  To that end,
I created what I call the Kitchen Sink Lua executable, which includes *all*
the modules we use at work.  At startup, I load all the luaopen_*() calls
into package.preload (these are listed in a luaL_Reg[] arraay) and add a
special loader to package.searchers to load the modules in Lua (that are
compiled via luac, compressed [1] and stored in the executable; the loader
finds the appropriate data and decompresses it when lua_load() is called).

  I do call luaL_openlibs() but upon reflection, my Lua state initialization
code (which is only called once---I don't create tons of multiple states)
could just be (sans error checking):

	static const luaL_Reg preloadtable[] =
	{
	  { "coroutine" , luaopen_coroutine },
	  { "table"     , luaopen_table },
	  { "io"        , luaopen_io },
	  { "os"        , luaopen_os },
	  { "string"    , luaopen_string },
	  { "math"      , luaopen_math },
	  { "utf8"      , luaopen_utf8 },
	  { "debug"     , luaopen_debug },
	  /* other pre-installed C-based modules */
	  { NULL        , NULL }
	};
	
	/*-----------------------------------------------
	; create our state with the bare minimum required
	; for a Lua state. 
	;-----------------------------------------------*/
	
	L = luaL_newstate();
	luaL_requiref(L,"_G",luaopen_base,1);
	luaL_requiref(L,"package",luaopen_package,1);
	luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_LOADED_TABLE);
	luaL_setfuncs(L,preloadtable,0);
	lua_pop(L,3);
	
  This (not checked, just off the top of my head) populates a Lua state with
the functions NOT in a module (like dofile(), pairs(), select, etc.) and the
package module, which I believe is the minimum required [2]. This does
assume that all scripts call require on basic modules like os and io. 
Preloading modules written in Lua is left as an exercise to the reader.  

  Would something like this work?
  
  -spc

[1]	I would get better compression by not pre-compiling the Lua code,
	but the code I have works, and the difference is not enough to
	actually worry about it.

[2]	I'm not sure how to make functions like assert(), getmetatable(),
	etc. avaialble via require().  It could be done I suspect, but would
	take a bit of code.