Re: Lua pattern similar to Python multiprocessing?

[Justin Cormack <justin@...>]

On Sun, Aug 12, 2012 at 3:55 AM, mattk <mksql@yahoo.com> wrote:
> In some Redis loading tests, LuaJIT 2.0 (beta) is performing quite nicely, at
> about 60% of the runtime of a similar single threaded Python script.
>
> Using Python's multiprocessing module,
> http://www.ngcrawford.com/2012/03/29/python-multiprocessing-large-files/
> chunking large text files , results in a significant performance improvement
> over a single process, splitting the works across cores. This is a
> multi-threaded approach, but multiple Python interpreters.
>
> I am assuming using the same approach in Lua / LuaJIT would perform even
> better, but as a Lua beginner, I have not found the correct approach.
> LuaLanes has been suggested. Essentially I am looking for a mechanism to
> create a master list of tasks / jobs, when are then dispatched to a number
> of worker processes < number of cores in the system.

The Python multiprocessing model is implemented on top of the posix
IPC mechanisms (shared memory, semaphores, message queues). You can
use all the primitives that make these up using Luaposix
https://github.com/rrthomas/luaposix/ but you wont get quite as nice
an interface out of the box, you will need to do a bit of work top
make an equivalent interface (like map etc).

Justin