Re: Question about LuaJIT's future FFI/Struct Extension

[Mike Pall <mikelu-1003@...>]

Duncan Cross wrote:
> I'll get to the point: How far away do you think you are from
> finalizing the API for the FFI/struct extension for LuaJIT?

I have a good grasp on what's needed for the data structure
definitions. This part will come first. So you'll be able to
create and use mutable buffers and low-level structs then. And you
don't need to learn a new type-specification language. You can
just cut'n'paste your C data type definitions.

Binding to foreign functions will come later. There are some tough
issues to solve, e.g. pointers vs. arrays or GC of callback
functions.

A general problem is that naming and binding differ substantially
between Lua and C/C++. It's a multi-stage, offline process in
C/C++. Any Lua-side API has to respect some of these conventions,
but still follow a dynamic and modular approach.

My approach is to first write some use cases, then design the API
that would fit them best and only then to implement it. Rinse and
repeat. It'll take a while to finalize.

> What is
> its priority relative to the other things you've got going on, now
> that the x64 port is complete?

I'm currently writing some more docs and will add an interactive
page for comparing the performance of Lua/LuaJIT for different
versions and on both x86 and x64. This is so I can prove to the
sponsors that the x64 port is on-par with the x86 port (which is
part of the sponsorship agreements).

Towards that goal I've already added array bounds check
elimination two days ago. That's essential for x64 performance
(e.g. SciMark LU), since Intel's Core2 lacks macro-op fusion for
compare+branch pairs in 64 bit mode (Core i7 rectifies that).

BTW: The SciMark LU score jumped from 922 to 1018 with ABCelim.
I've simulated the effect of the data structure extension and
predict the score to be ~1450, which is within 5% of GCC's score
of 1517.

I'll have to update the project-dependency graph before deciding
which sub-task I'll tackle next. The data structure extension is a
prerequisite for a few other things like string buffers. Which in
turn is a prerequisite for compiling many of the remaining
string.* and io.* functions. OTOH there are many NYI cases in the
JIT compiler that are in need of some attention. And there are
some pending redesigns and interesting optimizations, too.

> By "finalizing the API", I mean just a provisional description of all
> the functions it will include, not the actual implementation. Late
> last year, you posted an example of the kind of thing you had in mind,
> that involved ffi.def(), ffi.bindlib() and ffi.new() [1] - but
> cautioned that, at that point, you hadn't yet given it much thought.
> Have you had a chance to, since then?

I cannot guarantee any API at this time and certainly not the
definitions of any functions it will have. I might be coerced
into rambling about the included functionality though.

But let's turn that around: what would you (not just you, Duncan)
like to see in such an API? What would you like to do with it?
What might help to make it simpler and easier to maintain?
Real-life examples are preferred of course.

It's easier to bind C/C++ code to Lua than to other languages. But
from some mailing list postings, I get the impression it's still
like a black art for newbies. And it gets really messy once you
need anything not-so-simple: C++ OO vs. Lua OO, cross-language
inheritance, GC/memory management interactions, integrating
exceptions etc. ... I'm sure this list is incomplete.

> The reason I think it's important is this: Unlike other planned
> improvements to LuaJIT, this is an area where you cannot confidently
> write code today that will take advantage of the feature once it's
> implemented. If this extension will take a while to do (and possibly
> need another round of sponsorship?) I think it would be good if the
> API was known first, so people can start to get used to it, and maybe
> an unoptimised/imperfect placeholder implementation could be created
> (by somebody else) in the meantime. It would then be possible to write
> code for it immediately, make sure it's correct, and hit the ground
> running once the true JIT implementation is complete.

Maybe I'm misunderstanding, but I'd say it's a tad on the
speculative side to write code today against a not-yet-designed
and not-yet-implemented API for a not-out-of-beta implementation. :-)

--Mike