Re: Question about LuaJIT's future FFI/Struct Extension

[Mike Pall <mikelu-1003@...>]

I wrote:
> I'm currently writing some more docs and will add an interactive
> page for comparing the performance of Lua/LuaJIT for different
> versions and on both x86 and x64.

FYI: it's online now: http://luajit.org/performance.html

Doug Currie wrote:
> Integration of new data types implemented as userdata can be
> awkward: parsing, conversion to/from built-in types, comparison
> with built-in types, etc.

This is mainly an issue if you want to add operations on aggregate
types other than simple field/element access. You'll need that
glue anyway, but you may need to rewrite parts if you switch from
pure Lua data structures to low-level structs.

It's difficult to integrate types that are not subsets of existing
types. E.g. int64_t can only live as a boxed value -- the JIT
compiler may be able to unbox it, though. There should be no
auto-coercion into a lua_Number as this might lose precision.

Pavel Holejsovsky wrote:
> Maybe for beginning it would be nice to have possibility to mark  
> luCFunction as 'does not touch Lua state at all', therefore having no  
> LuaJIT-visible side effects.

But every C function bound with the standard Lua/C API does that,
or it wouldn't be able to get its arguments and return its
results. The stack setup for the arguments and the cost of the API
calls to fetch them are the main bottlenecks.

Declaring absence of other effects (e.g. on the GC) is probably
too hard for developers. E.g. most would probably be surprised to
hear that lua_tostring() might drive the GC forward if it gets
passed a number.

> Of source, full implementation of Mike's FFI proposal would make this  
> extension obsolete.

Yes, that's the way I plan to go.

Colin wrote:
> ...and then offer a simplified interface, that gives LuaJIT more
> freedom, for functions that are "pure" in this sense, i.e. that only
> want to access their arguments, and push some results, and otherwise
> leave the Lua state alone.

Actually my plan was to make this the default for any function that
doesn't get a lua_State pointer.

So you'd only need to add special declarations for C functions
that somehow call back into LuaJIT (e.g. using a hidden lua_State
pointer they've obtained before). Callbacks triggered from the
main event loop of a GUI come to mind. I think there's usually
just a handful of functions you'd need to tag this way (e.g. only
the entry to the event loop).

Knowing about other aspects of the called functions may also be
helpful for the JIT compiler. E.g. the GCC pure and const
attributes.

> "LuaJIT, here's a pointer of type 'int (*)( const char *, const char *
> )', give it global name 'foo' and generate the glue-code that gets the
> const char * out of the Lua state, and returns the returned int to its
> correct place, and you can trust me that the function will not touch
> the Lua state."

That example shouldn't be so hard. Mutable structures or non-const
declarations for string arguments pose more problems. And returned
pointers to arrays are unsafe for use, unless the array bounds are
known (there will be ffi.cast to regain some safety). Ok, so the
FFI is inherently unsafe, but figuring out a related crash in
JIT-compiled code is _really_ tough.

Lot's of little problems are awaiting me. I guess it'll take a
while before it groks all the declarations from something like
#include <gtk/gtk.h>.

--Mike