Macros and Token Filters Revisited

[steve donovan <steve.j.donovan@...>]

Hi all,

Since a few recent posts contained sentences like 'this calls out for
a macro' I thought I'd review the state of play for macros based on
token-filtering.  (Syntactical macros as implemented by Metalua are
technically superior, although harder to write;  this is a custom
compiler step generating bytecode).  Lexical macros using the
token-filter patch don't have an explicit compile step, since the
lexical stream within the Lua compiler is directly modified.  But you
do need a patched Lua.

lhf's token-filter patch was always intended as proof-of-concept;
I've done a somewhat more sophisticated version here (patch against
5.1.4):

http://lua-users.org/lists/lua-l/2010-02/msg00325.html

There is an explicit API provided for token filtering (no dependence
on global variables like FILTER) and C token-filters can be
dynamically loaded.  Although, only _one at a time_. In principle one
could chain token filters but it's awkward and could be rather slow;
the next iteration will explore the possibilities and trade-offs
involved.

It is true that a patched luac could be used to generate valid Lua
bytecode, which could then be distributed.  This is a little awkward -
Lua bytecode is not intended to be platform-independent.  (One of the
changes to Lua in Adobe Lightroom was to make the bytecode loader
platform-independent)

There is no doubt that some token-filtering facility would be very
useful for mainstream Lua, although it operates at a rather low level
of abstraction.  From the debates about the C/C++ preprocessor we know
the dangers and limitations, although macros are not as evil as
they're made out to be ;)  One of the dangers was creating private
non-standard languages (consider the original Bourne shell, written in
C macroized into Algol 68). But being able to easily craft a custom
syntax is sometimes a virtue, for instance if you were customizing Lua
to be an embedded DSL for your application.

steve d.