Re: Lua bytecodes and endian-ness

[askok@...]

> On 6/12/06, Luiz Henrique de Figueiredo 
> <lhf@tecgraf.puc-rio.br> wrote:
...
> > 0. Our philosophy regarding precompiled scripts is that 
> > loading them
> >   should be as fast as possible. That goal dictates the 
> > current
> >   implementation, which only works for native formats. 
> > Also, if the
> >   bytecode loader is to have a size advantage over the 
> > full parser,
> >   then the loader must be quite small.
> >
> > 1. It is very convenient to precompile scripts once and 
> > load them on
> >   multiple platforms. The solution is a universal 
> > bytecode loader,
> >   not a cross-compiler. However, such a loader is bound 
> > to be complex
> >   and possibly hard to test and maintain, if it caters 
> > for many platforms
> >   (but that's the whole point).

1. Hmm.. with the existance of both cross compiler, and 
universal loader, it would be easy to make a regression 
test suite that finds any bugs introduced automatically. 
About one days work?

> > 2. The endianness issue. If platforms differ *only* on 
> > endianness, then
> >   the modified lundump.c that I have posted is the 
> > solution. This goes
> >   against simplicity of the loader, but the convenience 
> > may offset this.
> >   Perhaps we could more official and distribute it in 
> > etc/.

2. Having the modified lundump.c in etc/ (or a patch) 
would actually be enough for the fink case I mentioned. 
Build code could then simply copy etc -> src before 
building, without needing to generate or fetch the patch 
from elsewhere.  That would do. :)   But I still think 
#ifdef'fed approach is actually better (basically, a 
question on how official you want to be with the 
approach).

> > 3. A cross-compiler can be useful if you cannot run luac 
> > on the target
> >   platform. The solution is a modified ldump.c suited to 
> > the target;
> >   this will keep the loader as simple as possible, which 
> > is the original
> >   goal.

3. Cross compiler has been requested frequently enough, in 
my opinion, that I fail to see why luac wouldn't be 
developed into that direction.  Possible in 5.2?  :)

> > 4. Lua does not depend on what ldump.c and lundump.c do, 
> > as long as the
> >   loader builds the correct internal data structures. 
> > You can replace
> >   ldump.c and lundump.c by anything you want, as long as 
> > they agree on
> >   the external format.
> >
> > 5. In the current implementation, this format has two 
> > levels: a
> >   structural level and a physical level. This should 
> > make it simple to
> >   replace the physical level, which only contains a 
> > handful of simple
> >   routines. Take ldump.c. The lowest physical level is 
> > implemented
> >   by DumpBlock and DumpMem. DumpMem is for data that 
> > could depend on
> >   endianness. So you can, say, write a dumper that saves 
> > files in a
> >   fixed endianness by simply rewriting DumpMem (which is 
> > currently
> >   a macro over DumpBlock). The other half of the 
> > physical level is
> >   implemented by DumpChar, DumpInt, DumpNumber, 
> > DumpVector, and
> >   DumpString. You can use a different external 
> > representation for
> >   integers, Lua numbers, and strings by rewriting some 
> > of those. The
> >   rest of the code in ldump.c implements the structural 
> > level. You can
> >   use a different structure by rewriting it. The loader 
> > is implemented
> >   in a similar way. LoadMem is for data that depend on 
> > endianness. The
> >   swap-aware loader that I posted earlier implements 
> > byte swapping
> >   in LoadMemand that's the only real change (except for 
> > testing the
> >   endianness of the file being loaded). Again, you can 
> > cater for
> >   different external representations by rewriting the 
> > corresponding
> >   routines in the loader.
> >
> > Finally, the header of precompiled scripts contain 
> > information about the
> > internal format. This can be used to make the necessary 
> > decisions for
> > complicaded loaders and bytecode transformers. The 
> > header also contains
> > a format number, which can and should be used by anyone 
> > writting a different
> > external format.
> >
> > All the modifications discussed above are simple to 
> > make, given a specific
> > goal or target platform. It's just that making all of 
> > them is too much to
> > include in the core Lua distribution.

Perhaps we need someone interested enough on this, to 
produce and maintain the cross-compiler, and other these 
issues?

Then again, some parts (the cross compiler, and having 
endianness read-in support) should at least be covered by 
the authors. Sorry if we're making your life harder, 
that's what the users are for!! :)   But, they are also 
fun, aren't they?  :P

-asko

> > If you want to modify ldump.c or lundump.c for a 
> > specific task, please feel
> > free to contact me (or post here) if you have any 
> > questions.
> >
> > --lhf
> > If you have any