RE: Properly binding C libraries using callbacks

["Thijs Schreijer" <thijs@...>]

> 
> I have a similar problem in my luvit project.  I'm wrapping an
> inherently async library that provides non-blocking I/O.  I ended up
> creating a helper that calls my lua callbacks from the main thread.
> 
> https://github.com/creationix/luvit/blob/master/src/utils.c#L7-37
> 
> As far as getting a reference to the lua state, the API I'm using
> always has a place for an opaque data pointer and I put in some struct
> that contains L and some other data (like a lua_ref integer) to get at
> the real data.
> 

I'm running into the callback issue as well, gave it some thought and came up with this; (warning; I only read about the c api, never used it, so might be rubbish...)


Async callback helper lib;

when the helper is loaded it registers a 'poll' function and places a value in the lua register. The key is the string, the value is a c pointer to an 'enqueue' function for reception of the async data.
so; [myhelperlib] = <address pointer to enqueue function>

the 'enqueue' function takes 2 arguments; 1) pointer to data, 2) pointer to a library specific 'decode function' and it should be protected/threadsafe. The helper will store the both pointers in a queue waiting for lua to collect it. 

Any library that has async callbacks to deliver data to the luastate, looks for the pointer to the 'enqueue' function in the register (when its 'openlib' function is called, not on the callback thread of course) and calls that function whenever it has data to deliver. It provides a lump of data (pointer 1) and a function (pointer 2). The function is a 'decode' function that is capable of decoding the 'lump of data' into lua data.

Whenever the 'poll' function (of helper) is called (from lua), the helper gets the first item from its queue, and calls the 'decode' function with 2 arguments; 1) the lump of data to decode, 2) the luastate. The 'decode' function is now called on the main luathread, so it can safely push the decoded return values onto the stack and return. (if there is nothing on the queue the 'poll' simply returns nil.)

As a notification mechanism; Whenever the helper receives data in its queue, it sends a UDP packet with only the number of items in the queue to 'localhost:someport' (possibly repeat this every x millisecs as long as there is data).

By convention the first argument returned would be a lua-function (a callback) which will be called with the other arguments as parameters.

End result;
 - uniform way of dealing with (async) callback data
 - helper is a standalone library, that can be used by multiple
   libraries at once
 - lua can call 'poll' for updates if its actively running
 - lua can use a socket select statement to wait for incoming 
   data, without a busy wait (udp works on windows and posix. 
   On posix pipes or filehandles may be used as well).


Could this work? Any caveats?

Thijs