Re: To all rock maintainers, take 3 (now with x64 supports)

[Stefano <phd.st.p@...>]

On 4 November 2015 at 04:42, Sean Conner <sean@conman.org> wrote:
>   Why do you still persist with the "contending module" errors?

Short answer:

I am working on ULua in my spare time among other projects.
Moreover I disagree that the added benefit justifies the added complexity.

Long answer:

Right now the whole package manger stands at around 1K lines.
And it's the first package manager I wrote, so probably it could be
25% smaller if I were to rewrite it.
Other savings could be done by outsourcing some components (semantic
versioning) or if Lua standard libraries were more extensive.
There is no need for a package database, and packages are nicely
organised in a package_name/package_version structure.
The metadata for each package is stored in a single short Lua file.
Nonetheless it supports multiple versions (loading the correct version
of each required module), multiple os, multiple arch, live updates of
the whole distribution, rollback in case of failures, updates, dynamic
libraries pre-loading, executable Lua scripts. It's even fast.

The feature you are requesting makes things way more complicated.
This is just a small sample of the difficulties you will encounter,
the devil is in the details:
Need to distinguish between foo.bar of package foo instead of foo.bar
of package foo.bar.
Need to check that updating package foo does not introduce a new file
that conflicts with a file of package foo.bar.
Need to discover all the relevant metadata that can be now anywhere.
Similarly,  documentation, examples, tests, executable scripts are now
in an arbitrarily nested structure.
We can no longer implement add/remove in terms of a single dir.mv
atomic operation.
And if you want to allow both jason4lua and luajson, welcome to
dependency conflict resolution!

All this for (it's really a single actually):
Being able to "namespace" using '.' (would using '_' be so bad as to
justify doing all the required work on the package manager side?)
Being able to split an already smallish codebase into even smaller
multiple packages.

I should also stress that his being a package manager it's not trivial
to test for safety and correctness.
Each added complexity introduces potential to new bugs which have the
ability to corrupt in an unrecoverable way an installed Lua
distribution.

>   So question:  What if I want org.conman.parsers.email?  Do you then drop
> org.conman.env?  What if you get a request for json4lua?  Someone needs both
> dromozoa-chunk *and* dromozoa-commons?

Do we really have to follow Java's nested naming route here?

Moreover, can I observe that we have a proliferation of incredibly
small / badly maintained packages offering a quite limited
functionality? Do we really need a separate rock for each single
hashing function ?

And I am curious as to what you propose with respect to jason4lua and
luajson, two modules who share:
require 'json'

Leaving the Lua ecosystem entirely in the hands of the Lua community
is at least partially to blame for this mess.
I do not share their approach on a number of aspects, but this is in
my view a reasonable take at managing a language ecosystem:
http://pkg.julialang.org/

And somehow they seem to manage just fine even without module invaders.

In any case, my MIT-licensed code is freely available at
https://github.com/stepelu/lua-pkg (ignore the README which is out of
date).

I am willing to be proven wrong and accept a patch that implements the
desired functionality provided:
1. it doesn't add significant complexity (300+ lines of code is not ok)
2. it is as robust and feature rich as the current implementation
(multiple versions for each package (correct loading), failures are
rolled back, no file is ever overwritten, live updates, ecc ecc)

I honestly believe such effort to be better spent in other ways (for
instance by automating a macdylibbundler [1] inspired approach to
create clib_* packages for external libraries on Linux, OSX and
Windows) but I will accept other people code contributions on the
discussed point with happiness!


Stefano

[1] https://github.com/auriamg/macdylibbundler