Re: To all rock maintainers, take 3 (now with x64 supports)

[Stefano <phd.st.p@...>]

On 4 November 2015 at 20:00, Sean Conner <sean@conman.org> wrote:
> It was thus said that the Great Stefano once stated:
>> On 4 November 2015 at 04:42, Sean Conner <sean@conman.org> wrote:
>> >   Why do you still persist with the "contending module" errors?
>>
>> Short answer:
>>
>> I am working on ULua in my spare time among other projects.
>> Moreover I disagree that the added benefit justifies the added complexity.
>
>   After some thought, I think I identified why this project doesn't sit well
> with me---it's not curated.  And because it's not curated, you provide a
> random collection of Lua modules.  Only it's not random, it *is*
> deterministic but some are excluded just because they have a conflicting
> module name who's project name sorts alphabetically prior to it (and for my
> modules you have it listed under the completely *wrong* name because of how
> you are treating module names).

There are two kind of packages for ULua: the ones imported from
Luarocks and the ones packaged directly for ULua.

Regarding the former, the goal is to make an effort to support the
largest number of cases.
Of course, as there are differences in the assumptions that Luarocks
make and that I make, so this can at times be difficult.
This doesn't mean that I am going to change the conventions that I
have chosen for my package manager to conform to Luarocks.

Also, I have no plans to act as a "curator". What I plan is to improve
my modules listing page by including additional information that can
help the reader to asses the quality of a module.
While these indicators (github stars, release dates, open issues,
tests included, tests passed) are by no means perfect, they are
clearly better than nothing.
It is sadly quite common experience to waste hours looking around the
web to decide which Lua module to use for even common functionality.

>
>   You wrote:
>
>> Unfortunately, a few rocks are not cooperative with 64 bit environments.
>> I would thus be glad if you, rock maintainers, would be so kind as to
>> assist by checking that your rocks are doing just fine.
>
> But my modules don't have issues with 64-bit environments (heck, I'm running
> on 64-bit Linux, 64-bit OS X and 64-bit SPARC) but they do have an issue
> with names.  And to me, I'm reading this as "please conform to my biases to
> package management."

Then clearly this didn't apply to you.

The "biases" are the conventions I decided to adopt for package
management having examined how things are done in a number of
different programming languages and package distributions.
I have already explained clearly the advantages they bring, so I will
not repeat myself here.
The webpage I linked is a clear example of what can be achieved if
some rules are adopted.
For instance, only by having conventions you can have things like
automated regression tests or the ability to update packages.

>
>> Long answer:
>>
>> Right now the whole package manger stands at around 1K lines.
>> And it's the first package manager I wrote, so probably it could be
>> 25% smaller if I were to rewrite it.
>> Other savings could be done by outsourcing some components (semantic
>> versioning) or if Lua standard libraries were more extensive.
>> There is no need for a package database, and packages are nicely
>> organised in a package_name/package_version structure.
>> The metadata for each package is stored in a single short Lua file.
>> Nonetheless it supports multiple versions (loading the correct version
>> of each required module), multiple os, multiple arch, live updates of
>> the whole distribution, rollback in case of failures, updates, dynamic
>> libraries pre-loading, executable Lua scripts. It's even fast.
>>
>> The feature you are requesting makes things way more complicated.
>> This is just a small sample of the difficulties you will encounter,
>> the devil is in the details:
>> Need to distinguish between foo.bar of package foo instead of foo.bar
>> of package foo.bar.
>
> Why?  Not not treat the module name as opaque?  I did that for a project I
> started back in 2014
> (http://lua-users.org/lists/lua-l/2014-05/msg00656.html) that I should
> probably pick up again.  But anyway, I treated the module name as opaque,
> with no structure.  Here's the latest examle of LEM file (which is nothing
> more than a ZIP file by the way):
>
> ...
>
>   While I could store both luuid's "uuid" and uuid's "uuid" modules in the
> same LEM, I don't have a way to specify which one to use.  That *is*
> something I would need to work on.  But as it stands, the LEM format can
> easily handle different operating systems, different architectures and
> different Lua versions (look closely---you'll see I have an LPeg 0.12 module
> for Lua 5.1 and for Lua 5.2).

That does not solve the issues I raised:
What about file conflicts across different packages (LEMs)?
How to perform updates when one package nest another one?
How to keep the complexity manageable while still achieving
rollback-on-failure safety?
Ecc ecc

>
>> Do we really have to follow Java's nested naming route here?
>
> No.  I could have use spc.net, spc.syslog, spc.signal, spc.errno, etc, but
> I had the example of Java [1].  It may not be the best, but at least it's
> *something*.  One problem with Lua modules is that the name used to load the
> module, say, require "uuid", doesn't necessarily match the name of the
> project---there are three different UUID modules listed in LuaRocks:
>
>         luuid                   provides "uuid"
>         uuid                    provides "uuid"
>         org.conman.uuid         provides "org.conman.uuid"
>
>   At the very least, my "project name" matches the "module" name.
> This does make it hard for a package manager to provide the requested "uuid"
> module (at least for the first two).

In my case I know what gets provided (indeed the Luarock package names
are essentially ignored), which is what results in conflicts.
Having the package name equal to the module name has, as you noted, advantages.

>
>> Moreover, can I observe that we have a proliferation of incredibly
>> small / badly maintained packages offering a quite limited
>> functionality? Do we really need a separate rock for each single
>> hashing function ?
>
>   The badly maintained I'll let slide, but small?  That's a problem?  And
> yes, I wouldn't mind a separate rock for single hashing functions.  Let me
> load just what I need.

I wasn't talking about your modules, I was talking about the state of
a non-negligible portion of the available rocks.

What I am stating is that the pervasive mentality in the Lua community
(reinvent the wheel just a bit differently, no backward compatibility,
no standard functionalities included, no conventions, 300 mini-modules
promptly abandoned) does not seem to have resulted in a solid
ecosystem with a good number of high profile packages.
I am not sure to what extent this can be argued against.

>
>> And I am curious as to what you propose with respect to jason4lua and
>> luajson, two modules who share:
>> require 'json'
>
>   I propose to you---what would you do if someone would like to use package
> manager to install json4lua?  Or a different example---someone needs
> numerical integration and thus, would like to use math-rungekutta but can't,
> because it "conflicts" with math-evol.

The thing is, right now math-evol is taking over math, which is
clearly bad (we already discussed about that).
Again, a problem that would be solved by having project name = module name.
Then, hopefully, something different from math would have been used.

>
>   Final parting words---if "conflicting modules" is too complex to handle,
> then please, remove "org" from your project.

I welcome your ides on how to handle it.
Not in the sense of storing them in a package. I am talking about the
package management logic here.

When I get around to implement rocks aggregation (meaning all your
rocks will end up under a single org package), you'll be free to opt
out if you want to.

Stefano