Building Modules
 |
||
|
Building Modules |
|
For instructions on how to build the Lua core yourself, see BuildingLua.
For instruction on how to write (as opposed to build) C extension modules, see BindingCodeToLua.
Building a C extension module generally means that you need to create a
shared library aka dynamic link library. This library gets loaded
at runtime by require() (new module system in Lua 5.1) or by loadlib()
(low-level call, available in Lua 5.0, too).
Static linking of modules against the core is also possible, but not recommended for generic Lua distributions (it makes sense for embedded systems, though).
Please feel free to correct or extend this list. -- MikePall
Note: Typically, Lua core header files (lua.h, lauxlib.h, etc.)
are not stored directly in the standard search path, but rather in a subdirectory.
You'll need to specify their location
to the compiler by adding -Ilua_header_file_directory to the compile
command, which is not shown in these examples. If your platform has pkg-config
and associated Lua data installed, you can get the needed compile flags as
follows (assuming Lua was installed under package name "lua5.1"):
LUA_CFLAGS=`pkg-config lua5.1 --cflags`
Although instructions specific to several platforms are provided below, on most
systems GNU libtool is available and will do the right thing while hiding platform
idiosyncrasies.
LIBTOOL="libtool --tag=CC --silent" $LIBTOOL --mode=compile cc -c module.c $LIBTOOL --mode=link cc -rpath /usr/local/lib/lua/5.1 -o libmodule.la module.lo mv .libs/libmodule.so.0.0.0 module.so
Note that cc is an alias for the "main compiler" on most systems. Plug in a
specific compiler and optimization setting as necessary. The -rpath value is
not critical, but the option is required to trigger a shared library build.
As a bonus, the above commands simultaneously produce a static version of the
library at .libs/libmodule.a. (Yes, it's compiled without -fpic.)
gcc -O2 -fpic -c -o module.o module.c gcc -O -shared -fpic -o module.so module.oUnfortunately
-fpic comes with a significant performance loss on x86.
You can omit this option on x86 (but not on some other architectures), but
then the shared library needs to be relocated upon loading it (basically
making a copy for every process). It depends on your usage patterns whether
this is really relevant (this is not a problem when you fork processes or
use native threads or non-preemptive approaches; it is a problem when
you exec lots of processes).
Add -fomit-frame-pointer on x86 for a significant performance gain
(unfortunately this prevents debugging on x86, too). Note: this does
not make any sense on other CPUs (x64 aka x86_64/AMD64/EM64T in particular).
E.g. gcc -O automatically enables -fomit-frame-pointer on all systems
where it does not interfere with debugging. Oh and x64 has no problems
with -fpic, too.
Avoid -fPIC since it comes with a big performance penalty on some RISC
architectures (Sparc in particular). The linker will warn you when you are
exceeding the limits for -fpic and then you have to switch to -fPIC.
It's very unlikely that you'll ever hit this limit with Lua modules, though.
Read more than you ever wanted to know about building shared libraries [here] (PDF, ~500K).
cc -O -Kpic -c -o module.o module.c ld -G -o module.so module.o
cc +O3 +Z -c -o module.o module.c ld -b -o module.so module.o
xlc -O2 -c -o module.o module.c xlc -G -bexpall -o module.so module.o
cc -O -Kpic -c -o module.o module.c ld -Bshareable -o module.so module.o
IMPORTANT NOTE: With the release of Lua 5.1.3 the module loader for Lua on Mac OS X has changed. It now uses the standard POSIX dlfcn API. So just follow the instructions above under dlfcn shared libraries with GCC. However, unlike GCC on other platforms, you must still link with the option '-undefined dynamic_lookup' to avoid getting undefined references to liblua. Whatever you do, don't link your extension against liblua.a! (See below: "Do Not Link Modules to the Lua Core Libraries")
It does not work for me (Mac OS X 10.4.9, powerpc-apple-darwin8-gcc-4.0.0). I get unrecognized option -shared. The instructions below work fine. -- lhf
"To properly build a module on OSX you need to either set a flat namespace and suppress undefined symbols OR have undefined symbols be dynamically looked up" -- jls
Unix/Linux equivalent:
gcc -bundle -flat_namespace -undefined suppress -o module.so module.o
OSX prefered:
gcc -bundle -undefined dynamic_lookup -o module.so module.o
Compiling and Linking
MACOSX_DEPLOYMENT_TARGET="10.3" export MACOSX_DEPLOYMENT_TARGET gcc -O2 -fno-common -c -o module.o module.c gcc -bundle -undefined dynamic_lookup -o module.so module.o
It took me a long time to figure this out, but the above link line is not sufficient on (at least) OS X 10.3, and perhaps 10.4. If you have mysterious crashes in your module, attempt to run it with DYLD_BIND_AT_LAUNCH=1 set in the environment. If the module now runs correctly, you probably have a problem with "lazy" initialization of data. Instead of setting the environment variable, add -Wl,-bind_at_load after the -bundle option. In paticular, modules implemented in obj-c, or that call obj-c have this problem, though it appears it can also be an issue with C++ as well.
Note: do not strip the dynamic symbols from the Lua executable
(use strip -x). liblua.a was not (and shouldn't be, see notes at end of page) linked into the module, so the bundle will need to find the lua_ API symbols in the executable when it is dynamically loaded.
Loading and Unloading
Modules that use obj-c have another problem, they CANNOT be unloaded, which lua attempts to do during shutdown. Symptom is something like this when lua is exiting:
The only fix I have found for this so far is to remove the __gc method from the _LOADLIB metatable:
or to hack loadlib.c so it never unloads C modules.
Module Naming Conventions
For consistency with other platforms the modules should be installed with the .so extension on Mac OS X, too, and this assumed by the default definitions of the package cpath.
You need a patched Lua, with support for oldskool CFM libraries.
For each module, create a matching foo.exp file with a single entry like:
luaopen_fooThen compile the foo module into a shared library "foo" (no extension)
A related page CreatingBinaryExtensionModules exists on this but is outdated. Possibly it should be merged into here.
lua51.lib to the input libraries (Link, Input).
note : make sure your main init function is exported with __declspec(dllexport) like this :
int __declspec(dllexport) MyModuleName (lua_State* L) { ... }
gcc -O2 -c -o module.o module.c gcc -O -shared -o module.dll module.o -Llua_dir -lluaXXReplace
lua_dir with the directory where the luaXX.dll resides.
luaXX.dll is either lua50.dll or lua51.dll, depending on
which Lua version you use.
Old GCC versions (pre 4.0) have problems with exception handling and stdcall's
when using -fomit-frame-pointer
([bug report]).
You can either upgrade or always use -maccumulate-outgoing-args with -fomit-frame-pointer.
Do not strip the relocation info from the DLL (use strip --strip-unneeded).
You may want to add -Wl,--enable-auto-image-base to the link command to
reduce DLL load times when you have many extension modules.
For notes on using LuaRocks under Cygwin, see [1] [2] [3] [4] .
LuaRocks installs Lua modules as self-contained packages (with dependency info) called "rocks".
Lua generally offers some level of API stability, but not necessarily ABI stability (binary compatibility). This means you need to recompile all your modules when you upgrade the Lua core even between minor versions (e.g. from 5.0 to 5.1). Be sure to always point your compiler to the correct include path (-I...) which contains the Lua core header files you want to compile against.
It's a good idea to recompile everytime when you use the development versions (the 'work' releases), because there are no API/ABI stability guarantees at all.
Although it may be tempting to link modules against the static
libraries containing the Lua core (lua*.a), this is not a good idea.
Essentially you are adding a copy of the whole Lua core (or at least large parts of it) to every module. Apart from being a waste of space this may lead to very difficult to diagnose problems. The different instances of the core code don't know anything about each other and may not necessarily be in sync.
In case you built a shared library containing the Lua core (*), please do not link any modules against it, too. I.e. do not specify the name of the Lua core library on the linker line (Windows DLLs are an exception). This makes a hard forward dependency where you really want a lazy backward dependency. Loading this module with a statically linked Lua interpreter would essentially drag in a 2nd Lua core and you get the same problems mentioned above.
Lua modules expect that the Lua core is already present before they are loaded.
This works because the Lua core symbols are exported globally (e.g. with -Wl,-E
for GCC on most ELF systems -- see the Lua Makefiles).
Related caveat: this is also why you shouldn't link an application with the dynamic
library containing a Lua module. This usually does no harm, but doesn't do what you
want because the luaopen_foo() function is never called. Modules are to be loaded
by require() and not via hard dependencies in the image (package dependencies are ok).
(*) Compiling the Lua core with -fpic on x86/POSIX platforms has even more serious
performance drawbacks than doing this just for modules (see above). The recommended
way (shown in the Lua 5.1 Makefiles) is to link the Lua core statically and export
all symbols (enabled with -Wl,-E or by default on Solaris and others).
The above build instructions for Windows assume that you have compiled the Lua core in a specific way:
lua.c, luac.c, print.c and build luaXX.dll (i.e. either lua50.dll or lua51.dll).
lua.exe from lua.c and link it to luaXX.dll.
It's important to compile and link both the standalone executable and
every module to the same lua header files and the same
luaXX.dll.
lua51.dll is arguably preferable. There has been some confusion here caused by Lua Binaries. The Makefile in the original Lua 5.1 distribution builds Lua with a DLL named lua51.dll. Lua Binaries 5.1 instead named this file lua5.1.dll. More recent versions of Lua Binaries, following recognition of the confusion caused, include a proxy DLL named lua51.dll that forwards calls to lua5.1.dll (see LuaProxyDllThree), thereby allowing Lua Binaries to work with Lua extension DLLs that use either linkage. Most other Lua distributions that expect lua51.dll (including LuaJit [12]) lack a proxy back to lua5.1.dll, though you could add the proxy yourself if needed. Therefore, Lua extension DLLs that link to lua51.dll could be considered the most compatible with all distributions. The proper naming has been discussed a numbers of times as shown in the links below. In Lua 5.2.0-work, LuaBinaries uses lua52.dll, just as the official source tarball does [5].
Almost every compiler for Windows seems to bring its own
standard C library to link against (e.g. the MSVC*.DLL variants):
It is strongly recommended that all executables and DLLs working together in a single process share the same C library. This avoids potential problems such as crashes and difficult to diagnose behavior that can occur when passing certain data structures between different C libraries (see MSDN article below). The Lua core carefully avoids such passing (e.g. Lua manages internally all memory and file handle allocation/deallocation), though you might still run into pitfalls if not careful (examples anyone?).
It's also possible that an extension DLL need not link against any C library if all it needs is available in Lua DLL, system DLLs, and statically linked code.
Here's some background information:
By default, MinGW builds against the old runtime (msvcrt.dll) but can be persuaded to link against msvcr80.dll using -lmsvcr80. However, you may have trouble with non-trivial extensions, because the import library supplied with MinGW has got some import names wrong. For instance, _findfirst is actually exported as _findfirst32, etc. The solution is straightforward, although a bit of a hack: link directly against the DLL and rename the offending functions.
Here is a makefile that can be used with MinGW with Lua for Windows:
Here is the simple makefile I used for rebuilding lfs; note that no import libraries are needed, only the Lua header files.
LUA_INCLUDE= d:\stuff\lua\src
LFW= c:\Program Files\Lua\5.1
LUA_LIB="$(LFW)\lua5.1.dll"
RT_LIB="$(LFW)\msvcr80.dll"
lfs.dll: lfs.c lfs.h
gcc -shared -I$(LUA_INCLUDE) lfs.c -o lfs.dll $(LUA_LIB) $(RT_LIB)
And here is the list of renamed symbols:
#define _ctime _ctime32 #define _difftime _difftime32 #define _findfirst _findfirst32 #define _findnext _findnext32 #define _fstat _fstat32 #define _ftime _ftime32 #define _futime _futime32 #define _gmtime _gmtime32 #define _localtime _localtime32 #define _mkgmtime _mkgmtime32 #define _mktime _mktime32 #define _stat _stat32 #define _time _time32 #define _utime _utime32 #define _wctime _wctime32 #define _wfindfirst _wfindfirst32 #define _wfindnext _wfindnext32 #define _wstat _wstat32 #define _wutime _wutime32
(In my projects lately, I follow the above list of functions to alias with macros at compile time, and use a link line that ends with:
Where LUADLL is a fully qualified name of lua.5.1.dll and MSVCR80 is the fully qualified name of MSVCR80.DLL. Since I use LfW, I find both of those DLLs in its installed tree, based off the LUA_DEV environment variable it defined. The result is a dependency tree that has two implicit loads of MSVCR80.DLL, and only loads MSVCRT.DLL due to MSVCR80's need. If I leave out libgcc.a or otherwise move it after either reference to MSVCR80, then I find unhealthy dependencies on MSVCRT.DLL.
Your mileage will vary, so I strongly recommend checking DLL dependency carefully with Dependency Walker or the equivalent before releasing a binary. --RossBerteig)
Given that it is not always easy or practical to ensure that all modules in a process use the same C run-time libraries, it's noteworthy that if your C extension DLL doesn't call any ANSI C functions, it doesn't need to link to any C runtime, thereby avoiding the issue on its part. For example, let's write a very simple module that wraps the Win32 MessageBox[8] function:
#include <lua.h>
#include <lauxlib.h>
#include <windows.h>
BOOL APIENTRY DllMain(HANDLE module, DWORD reason, LPVOID reserved) { return TRUE; }
static int l_messagebox(lua_State * L) {
const char * s = luaL_checkstring(L, 1);
MessageBox(NULL, s, "messagebox", MB_OK);
return 0;
}
__declspec(dllexport) int luaopen_messagebox(lua_State * L) {
lua_pushcfunction(L, l_messagebox);
return 1;
}
Since there are no ANSI C functions (but rather only Win32 and Lua functions), we can omit the C run-time from the linker options, as shown below for various compilers:
# GCC MinGW (under Cygwin -mno-cygwin) gcc -mno-cygwin -O2 -nostdlib -shared -I<lua_include_path> -o messagebox.dll messagebox.c -luser32 <lua_bin_path>/lua51.dll -Wl,-e,_DllMain@12 -Wl,--dll -nostartfiles # MSVC++ 2008 cl -O2 -LD -I<lua_include_path> messagebox.c <lua_lib_path>/lua51.lib user32.lib -link -nodefaultlib -entry:DllMain
The resultant DLLs are only about 3 KB. You can confirm with dumpbin or objdump that the DLLs only link to user32.dll (MessageBoxA) and lua51.dll (luaL_checklstring and lua_pushcclosure).
Now, what if your C extension DLL needs to do something ANSI-C-y like allocate memory or write to a file? One option to to use the Win32 API functions (e.g. HeapAlloc [9] or CreateFile [10]), which is what the C library itself does (you can confirm by statically linking a program to the C library and looking at the imports). This isn't very portable if your program needs to compile on non-Windows operating systems. Another option is to rewrite the C runtime libraries you need in terms of plain Win32 API calls (see [tlibc - Tiny C Runtime Library]). Another alternative is to access these functions indirectly via Lua. For example, call [lua_newuserdata] to allocate a memory block or do lua_getglobal(L, "io") to obtain a reference to Lua's I/O library.
The third option is to use the ANSI C functions but, yes, dynamically or statically link to a C run-time that might be different from the C runtime used by other modules in the process. This is not normally recommended because of the possible pitfalls described in the MSDN article [11]. However, it can be done by giving careful attention to those issues, so there's nothing inherently incorrect in doing it. For example, it should be safe to do
...
#include <stdio.h>
static int l_messagebox(lua_State * L) {
char * s = malloc(1000);
if (s) {
sprintf(s, "%e", luaL_checknumber(L, 1));
MessageBox(NULL, s, "messagebox", MB_OK);
}
free(s);
return 0;
}
...
# GCC MinGW (under Cygwin -mno-cygwin) - links to msvcrt.dll
gcc -O2 -mno-cygwin -shared -s messagebox.c -o messagebox.dll
-I<lua_include_path> <lua_bin_path>/lua51.dll
# MSVC++ 2008 - statically linking to the C runtime.
# Note: The resultant binary is about 66 KB when statically linking.
cl -O2 -LD messagebox.c -I<lua_include_path>
<lua_lib_path>/lua51.lib user32.lib
Both resultant binaries do work in a version of Lua compiled against msvcrt90.dll.
Note that the C runtime often needs to be initialized. Typically, the C runtime defines its own entry point (e.g. DllMainCRTStartup), which initializes the C runtime library and in turn calls the user's entry point (e.g. DllMain), so this is normally taken care of for you. However, if you override the DLL entry point, the C runtime won't get properly initialized.
In general I don't recommend to strip binaries. With modern binary formats (such as ELF) and virtual memory systems all debugging info is stored in data pages that never get mapped into memory. Thus all debugging information only takes up space on the hard disk (which is usually abundant), but not in memory. The situation may be different for embedded systems, though. I've only added warnings for systems where you need to be careful with stripping.
The settings listed above are conservative defaults. Of course you can go
crazy with -O3 -march=xyz and various -f<something> flags. Do it
if your application really needs it. Don't bother, if you have no need.