parallelizing thousands of tests

[nr@... (Norman Ramsey)]

I've used Lua to create some infrastructure that runs unit tests.
In a typical run we run a couple hundred tests against maybe 30
programs, so a total of around 6000 tests.  The Lua code has almost no 
overhead; we're mostly launching a lot of processes.  Result: I can't
keep even one CPU core busy.  We have an embarrassment of parallelism,
but I can't figure out which of the many packages I ought to try.

Here is the loop I wish to parallelize:

  for tid, tests in pairs(tests) do  --- outer loop
    local results = { }
    matrix[tid] = results
    for i, test in pairs(tests) do   --- middle loop
      if test.valid then
        results[i] = { }
        local results = results[i]
        for sid, bin in pairs(binaries) do -- inner loop
          local outcome, witness = run_test(test, bin)
          results[sid] = { outcome = outcome, witness = witness }
        end
      end
    end
  end

I'd be perfectly happy to try running the outer and middle loops
sequentially but to spawn a separate thread to run the inner loop in
parallel.   That would result in 20 to 30 threads in flight at once,
which is probably enough to keep 8 cores busy without an embarrassment
of parallelism.

Can anyone suggest which of the many parallel packages for Lua might
be most suitable for parallelizing this loop?


Norman Ramsey