Re: how to exploit multicore cpu w/ Lua?

[Dimiter 'malkia' Stanev <malkia@...>]

On 3/30/2012 12:45 AM, Valerio Schiavoni wrote:
> Hello,
> what is the suggested way to exploit a multicore cpu when using Lua?
> Is Lua-Lanes the suggested way to do so ?
> The problem with that is: our codebase doesn't rely on it, and this
> doesn't seem to be a swap-in replacement.
>
> The ideal scenario I had in mind was the ability to have a modified
> LuaVM that bundles LuaLanes behind the scenes, making it invisible
> to the developers. Legacy code would not need any modification (calls to
> coroutines stay unmodified): the VM transparently
> wraps those calls and redirect them to the appropriate LuaLanes calls.
> Is this reasonable ? Does this "LuaLanesVM" exists already ?
>
> Are there other ways to exploit multicores ?

Spawning multiple processes, or at least multiple instances of lua 
states, that communicate through some ipc.

And/Or for more data-heavy process (image filtering, etc.) use language 
like OpenCL, CUDA - and like lua - most of the implementations allow 
compilation on the fly, so iteration, code generation and ability to 
transport the code on another machine/process is not lost.

Or maybe this silly idea - Limited form of OpenMP:

As in openmp, there would be some kind of directive, or just special 
keyword "formp" that works only with integers (probably tricky). Also 
before the loop, all garbage collecting is stopped, and afterwards 
restored. Not only that but anything that would change the lua objects 
memory would cause a break back to some kind of serial mode recovery, or 
just errors out.

local array = ffi.new( "float[?]", n_elements )
-- Here garbage collecting stops
-- The lua_State is replicated, and "i" somehow is initialized for each 
thread to start from (K/T..(K+1)/T-1) for each thread, where K < T, and 
K is each thread, and T is the number of threads
-- Or like OpenMP with dynamic scheduling, the K is calculated in 
different fashion for each thread...
formp i=0, n_elems-1 do
  array[i] = somecalc(i)
end

Here instead of luajit's ffi types, some lua types can be put in places 
- native arrays which don't change, and multiple threads can read/write 
to them.

Now off course somecalc(i) should not also do any changes to the memory 
layout that would break lua.

> Thanks for your comments.
>
> Valerio