Re: helper threads layer

[Mark Hamburg <mhamburg@...>]

Here is a short overview of the system we're using in Lightroom. It isn't as
simple as what Javier has proposed but it shares some similarities. It also
has evolved over time and has not had the opportunity to undergo as many
complexity-reduction passes as it probably deserves.

Concept 1: Multiple universes

We run multiple Lua universes that can generally function independently
without regard to a mutex.

Concept 2: Universe-to-universe copying

We implemented copying for a subset of Lua data structures between
universes. It is up to the caller to insure that neither universe is
attempting to execute other code at the time.

Concept 3: Transit universe

We implemented a primary transit universe which is wrapped with a mutex and
that is intended for communication between universes. (But see below.)

Concept 4: Transit queues

We implemented queues with values in the transit universe and appropriate
conditions and mutexes for reading and writing. Transit queues are capable
of being referenced from Lua and are reference counted so that they can be
referenced from multiple universes safely. (Note: In retrospect, we could
have just implemented each queue with its own universe.)

Concept 5: Spawning universes

We implemented a call which takes a Lua function without upvalues and for
which a default environment is acceptable and provided a way to create a new
Lua universe, load that function, and start it running with a set of
parameters carried over again via the intra-universe copying mechanism. In
particular, we can pass it queues to wait on for input and to write to with
results.

These required small bits of native code to implement, but the result is a
system in which most of the structure can be wired up in Lua.

Code running in other universes should work on limited amounts of data since
we want to limit the amount of data to copy between universes, but there are
plenty of tasks where we can take a small description, go churn away for a
while, and come back with a reasonably small answer.

Mark