Re: recognizing whether a caller uses function result

[Frank Mitchell <frank_mitchell_us@...>]

Sorry for any confusion. My mail service was dumping most of the Lua-L list into a spam folder, so that and being a little slow on the uptake meant that it took me a bit to figure out what your Redis application was doing and why.

As for the dream project I briefly described: I want to create a configurable server with Lua interpreters in separate threads communicating with each other and with some components written in C through "message channels". I drew the basic idea from Communicating Sequential Processes[1]. Much like Erlang, Lua processes would activate when they received a message, then either pass a response back (like a micro-service) or send other messages along other channels to other Lua (and non-Lua) processes. The C-written portions, besides the threading and message channel components, would include a non-blocking I/O module using `select()` that translates JSON documents (for example) into messages to the (mostly) Lua-written processing components.

As you said, it's important not to spawn more threads than the machine has cores, otherwise the extra threads will only reduce throughput. I'd probably have to create a pool OS threads and allocate application Processes among the actual threads. (Or else I'd use some user-space threading library to essentially do the same thing and hope it's better at scheduling than I am.) Also, from a similar application I had a hand in building, it's best not to put too many stages, queues, channels, etc. in and have each module -- or standard services between modules -- do as much work as possible.

For example, if this thing were emulating a Message Queue, I'd probably put in one C nonblocking I/O stage to manage external connections, one Lua stage to put messages into the chosen persistent queue and send back acknowledgements to all parties involved, a third Lua stage to poll a requested queue and forward a message back through the I/O stage, and a fourth Lua stage to receive acknowledgements from the end-consumer and delete consumed messages from the queue. Maybe if I had enough cores I'd create multiple instances of each, but I'd *measure* before optimizing, because the more stages in the pipeline and the more components competing for cycles the slower this thing will get. That's just Multi-Threading 101.

There's no specific goal in mind for this whole architecture, apart from saying I've done it.  That said, if people are writing micro-services and whole servers in (mostly) JavaScript, there's no reason they can't use Lua.

At a startup I used to work at, where we were building Message-Oriented Middleware, we started in broad strokes with an architecture like this. Unfortunately a lot of things went wrong with the implementation, and not just because we were using Java. Pro-tip: when your twenty-something senior software architect wants to build something based on an article he read in a magazine (it was an early version of Spring, btw), ask why we're not just using the thing described in the magazine.


Frank Mitchell

[1] https://en.wikipedia.org/wiki/Communicating_sequential_processes


P.S. Also, when your task is to build a Java Message Service, first build a Java Message Service, not a general, flexible architecture with which to build any sort of service including a Java Message Service. Alas this axiom prevents me from building my Lua-CSP Frankenstein's Monster until I can find a real purpose for it. I should probably just learn Erlang and AWS instead.