Bundling Lua code in your (Mach-O) executable.

[David Jones <drj@...>]

I recently did some work on Lua MIDI, http://sourceforge.net/projects/ 
luamidi/ , and part of work was bundling the Lua code directly into  
the executable.

Part of the build process for Lua MIDI builds an executable and  
stitches a .lua file into the executable so that the executable is  
standalone.  Part of the application is written in Lua but the .lua  
files do not need to copied separately; the entire application can be  
distributed as a single binary file.

There are plenty of ways of doing this (for example translate  
the .lua files into .c files that have the contents in a C string), I  
think mostly the reason why I chose this way is that I happened to be  
reading about the Mach-O loader whilst I was needing this feature.

Because I use the Mach-O loader interface this technique only works  
on OS X; though the general principles will be similar on other  
operating systems, most of the work is in the platform-specific  
details.  Since the work was non-trivial I thought I'd jot down a few  
notes in case anyone else wanted to do something similar

The technique works by creating a Mach-O Segment Searcher, MOSS.   
MOSS has the following features:

- Include one extra C file, the MOSS implementation, and call one  
function to enable MOSS in your application.
- require'foo' will search for foo in the Mach-O executable.
- Embed Lua code in your Mach-O executable at link time by adding a  
few options ("just link-and-go").
- External foo.lua file will override internal foo, easing the  
editing and development of pure Lua modules whilst the application is  
running (no relink required).

Anyone who wants the gory details (and code!) i urged to look at code/ 
lmoss.h and code/lmoss.c in the recent Lua MIDI release: http:// 
sourceforge.net/project/showfiles.php?group_id=131184 .  I didn't  
really think it was worth unbundling.

moss_loader is the function installed in the package.loaders array.   
It allows modules to be found in the Mach-O executable.  moss_loader  
works by grovelling over the structures in the Mach-O executable  
header, using the not-very-well-documented structures in <mach-o/ 
loader.h>.  Here's a typical extract of the code (this finds the  
command segment called "Lua", it is used when MOSS is initialised):

  const struct load_command *command;
  const struct segment_command *segment;
  int i;

  command = (void *)(moss_header + 1);
  for(i=0; i<moss_header->ncmds; ++i) {
    if(command->cmd != LC_SEGMENT) {
      goto nextCommand;
    }
    segment = (void *)command;
    if(memcmp(segment->segname, "Lua", 4) != 0) {
      goto nextCommand;
    }
    /* Found a segment named "Lua". */
    moss_section = (void *)(segment + 1);
    moss_segment = segment;
    return;

nextCommand:
    command = (void *)(((char *)command) + command->cmdsize);
  }

Later, when require is called, the moss_loader function will search  
this segment for a section of the required name.

Conveniently the Mach-O file structure allows for a named segment to  
contain several named sections.  So each Lua file can go in a section  
of its own name (foo.lua goes in section foo, say), and all those  
sections go in a single segment called "Lua".

Here's how you include a file in a named segment at link time  
(further reading in ld(1)):

gcc -o lua  lua.o liblua.a -lm -arch ppc -arch i386 -g -sectcreate  
Lua lmidi ../../code/lmidi.lua ../../code/lmidi.o ../../code/lmoss.o - 
lreadline -framework CoreMIDI -framework CoreFoundation -framework  
CoreAudio -framework Carbon

Notice the -sectcreate option, "-sectcreate Lua lmidi ../../code/ 
lmidi.lua" includes the contents of the file "lmidi.lua" in the  
section named "lmidi" in the segment named "Lua".  MOSS can now find  
this Lua file if you use "require'lmidi'".

The "just link-and-go" feature means that in principle an entire  
application consisting of Lua and C code can be built with one  
invocation of the compiler:

cc -sectcreate Lua foo foo.lua [repeat-for-each-Lua file] *.c

Cheers,
 drj