Package proposal modification proposal

[Diego Nehab <diego@...>]

Hi,

The Lua 5.1 package proposal has been out for a while. Since its
release, it has changed little and I've usually been one of the first to
come to its defence whenever changes were proposed. I still think it is
simple and general enough that pretty much anything can be accomplished
one way or another (much like Lua itself). Nevertheless, I have failed
to find a satisfactory solution for the following problem, without 
resorting to a small change in the system (no colateral effects).

This is a proposal and request for comments.

Assume the following general scenario. Package "foo" has two
sub-modules, "bar" and "baz". It also provides some funcionality in the
top level, so that it makes sense to require"foo", "foo.bar", and
"foo.baz" independently.  Let's assume further that "bar" is implemented
in C, and that "baz" is implemented in Lua. (*)

The standard way to distribute something like this would be to provide
files foo.so, bar.so and baz.lua, and install them as follows:

    <lua-root>/foo.lua
    <lua-root>/foo/bar.so
    <lua-root>/foo/baz.lua

Depending on the situation, it might be preferable to package all three
files into one single dynamic library, zip file, whatever. One important
reason is that it makes things easier to deploy.  Another reason is that
on Windows it is expensive to load a DLL, but it is cheap to search for
an entrypoint in a loaded DLL. In othe words, the layout below might be
prefered:

    <lua-root>/foo.so

Now, the library writer probably has his own idea of what is the best
way to deploy his/her library. His views might or might not agree with
those of whoever else is packaging a distribution that uses that
library. It would therefore be good if little or no change was required
in library code when the layout is changed.

Conversely, it would be advantageous if little or no change to the
"initial state" of the interpreter was required to support different
layouts. This would encorage application developers to adopt the package
proposal even if they have no intention of releasing their internal
libraries.

The problem with the current system is that there is nothing foo.so can
do to prevent require"foo.bar" and require"foo.baz" from failing, now
that bar.so and bar.lua don't exist anymore in the search path.

Populating package.preload means we would keep foo.so in memory
even when it's not needed. It would also require us to change the
"initial state" of the interpreter somehow.

My suggestion is simple. Let require"foo.bar" expand into (almost the
same as) require"foo" followed by require"foo.bar". Accordingly,
require"foo.bar.qux" would expand into the sequence require"foo",
require"foo.bar", require"foo.bar.qux". The difference between a direct
invocation of require"foo" and the implicit invocation produced by
require"foo.bar" is that the latter doesn't complain on failure: it just
moves on.

With that, all we need is luaopen_foo inside foo.so to populate
package.preload with loaders for everything that it wants to export
(including perhaps itself). luaopen_foo (already) receives the name of
the package being required, so it knows what the situation is: direct or
implicit invocation. It could even populate package.preload in a lazy
fashion, when individual components are requested. luaopen_foo only has
to load the "foo" package itself when require"foo" is called directly,
not implicitly. By returning "false", luaopen_foo (already) can prevent
caching.

The only colateral effect I can see is that for old libraries that do
not know they will be invoked for sub-modules, "foo" will be loaded when
require"foo.bar" is invoked, even if "foo.bar" doesn't depend on it.  I
don't think this is a problem. Especially because newer libraries will
be able to avoid this if they want to.

Best regards,
Diego.

(*) This mix happens in LuaSocket, of course. Don't think I am being
selfish, though: as libraries grow in number and size, I believe this 
will happen more and more often.