Source Preprocessing

wiki

• http://perldoc.perl.org/perlfilter.html
• http://en.wikipedia.org/wiki/Preprocessor
• http://search.cpan.org/~jhi/perl/lib/Filter/Simple.pm
• http://dev.perl.org/perl6/rfc/264.html
• http://www.perl.com/lpt/a/663

Source filtering can be used to extend the syntax of a language. It some cases it can be used to improve the efficiency as well.

Lua provides a loadstring functions (similar to eval in other dynamic languages) that can be used to compile Lua code on the fly. This allows a Lua program itself to in some ways filter itself or other code in-process rather than via a separate preprocessor (e.g. as in the C preprocessor).

In the below example, we do a benchmark comparison between a typical method of doing vector multiplication and that implemented using source filtering macro-expansion. This mostly for illustration, and it's not necessarily recommended.

local lua = [[
  -- typical implementation (without filtering)
  local function cross_vector2(u, v, result)
    local u1, u2, u3 = u[1], u[2], u[3]
    local v1, v2, v3 = v[1], v[2], v[3]
    result[1], result[2], result[3] = u2*v3-u3*v2, u3*v1-u1*v3, u1*v2-u2*v1
    return result
  end

  -- test and benchmark
  local function benchmark(func)
    local t1 = os.clock()
    func()
    local t2 = os.clock()
    print("time:", t2-t1)
  end
  benchmark(function()
    local vector u = 1, 0, 0
    local vector v = 0, 1, 0
    for n = 1,5000000 do
      cross_vector(u,v,u)
      --print("DEBUG: u=" .. stringify_vector(u))
    end
  end)
  benchmark(function()
    local u = {1, 0, 0}
    local v = {0, 1, 0}
    for n = 1,5000000 do
      cross_vector2(u,v,u)
      -- print("DEBUG: u=" .. table.concat(u, ', '))
    end
  end)
]]

-- source filtering implementation
lua = string.gsub(lua, "local%s+vector%s+(%w+)%s*=%s*(%w+)%s*,%s*(%w+)%s*,%s*(%w+)", "local %1_1, %1_2, %1_3 = %2, %3, %4")
lua = string.gsub(lua, "cross_vector%s*%(%s*(%w+)%s*,%s*(%w+)%s*,%s*(%w+)%s*%)", "%3_1, %3_2, %3_3 = %1_2*%2_3-%1_3*%2_2, %1_3*%2_1-%1_1*%2_3, %1_1*%2_2-%1_2*%2_1")
lua = string.gsub(lua, "stringify_vector%((%w+)%)", "(%1_1 .. ',' .. %1_2 .. ',' .. %1_3)")

-- source filter
print("DEBUG[\n" .. lua .. "]")
assert(loadstring(lua))()

Results on one system:

time: 0.937

time: 4.078

The above could be made more robust. In particular, additional care should be taken to ensure that the source filtering doesn't occur in unwanted places. For example, in the remote case that your program had

print("cross_vector(u,v,u) = ", cross_vector(u,v,u))

you probably don't want source filtering to occur inside the string. To handle this, your source filter will need to skip over strings. This may require a full Lua parser (e.g. [LuaParse]), or you may even get away with a partial lexer (e.g. LuaBalanced). Luckily, Lua is relatively easy to parse compared to other languages.

Interaction issues may occur if you attempt to use two independent source filters at the same time.

Source filtering is related to code generation. For example, see the usage in LuaList:2006-09/msg00798.html.

--DavidManura