Re: More about packaging

[Mike Pall <mikelu-0406@...>]

Hi,

> For libraries that have both a C part and a Lua part, the thing gets a
> little messier. Our smtp module needs help from C, so we have smtp.lua and
> smtp.so. I wonder what is a clean way of loading such a
> module, that works both statically and dynamically when the user
> script calls require"smtp"...

The Pythonistas have solved this by prepending an underscore to the name
of the C part if it conflicts with the Python part. So you often see stuff
like this:

Module foo.py:

from _foo import *      # import the C module and copy the namespace
...                     # add own objects to the namespace

[Just FYI, in Python there are three ways to import modules:

 import foo
 from foo import x [as a], y [as b],...
 from foo import *

 The first one binds the imported module under the same name to the local
 module namespace. So you need to refer to "foo.member" to get access to
 a member of the imported module.
 The other ones bind selected members of the module under selected names
 to the local module namespace. So after 'from foo import member as mymember'
 you can refer to "mymember" (but not to "foo" nor to "member").
 Internally all imports are cached based on the name given to the import
 statement (works just like _LOADED in Lua).

 And no ... we certainly do not want to copy the whole complexity for Lua!
]

In fact the Python import statement transparently either reads/compiles
Python code or dynamically loads C libraries. This way the user of a module
does not have to worry about that issue. Yes, I want that for Lua, too!

We should specify a set of conventions to be followed by developers,
so we get more interoperability. Ok, let's start a list:

Conventions for library developers:

- Libraries should not pollute the global namespace.

- Libraries should return the library namespace (a table) that holds
  all objects (functions, strings, other tables, ...) defined by it.

- A Lua library should return the library namespace at the end of
  initialization.

- If a C library is dynamically loaded, the initializing function should
  return the library namespace

- If a C library is statically bound to the application it should
  assign the library namespace to _LOADED[libname].

- In case of a naming conflict between a Lua library and a C library that
  is required as part of the Lua library, prepend an underscore ('_')
  to the name of the C library.
  Load the C library as part of the initialization process of the Lua
  library. Copy the objects from the C library namespace to the Lua library
  namespace as needed. Users of your library should not need to know
  whether they are dealing with an object created by the Lua or the C part.

- Libraries requiring other libraries must take care to import them
  to a local symbol, to avoid polluting either the library namespace
  or the global namespace. Avoid circular dependencies at all cost.

Conventions for application developers:

- Always import a library to a local symbol. Yes, this means you have to
  import the library into every Lua source file that needs to access it
  (the imports are cached, so efficiency is not an issue). Use consistent
  naming among all your source files.

- If possible use the same name for the symbol and the library to reduce
  confusion for an innocent reader of your code.

TODO: Add some conventions for installation and packaging of libraries
      and applications.

So if we follow this model then we need an addition to the Lua core:

- A way to load either a Lua or a C library with a single function.
  The function should internally use _LOADED, require() and loadlib()
  and take care of platform specific issues.
  We could either use LUA_PATH for both C and Lua libraries or add
  LUA_DYNPATH and define a precedence rule.
  We could either extend require or define a new function (what would
  be a proper name?).

All of this is IMHO of course. Please, everyone share your thoughts with
the list on that issue!

Bye,
     Mike