Re: Macros and Token Filters Revisited

[David Manura <dm.lua@...>]

On Wed, Jun 16, 2010 at 12:06 PM, Luiz Henrique de Figueiredo
<lhf@tecgraf.puc-rio.br> wrote:
>> One thing that doesn't get carried over with a source translation
>> would be original line numbers.
>
> Why not? Just expand tokens with the same line numbers as the tokens that
> trigger the expansion.

That can be a useful approach, but it might not be a general solution
if the lines get rearranged:

macro square(x) (x*x)  -- (1)
print square(x)     -- (2)
  unless bar  -- (3)

if not bar then   -- (3)
  print (x*x) end  -- (1,2)

unless you transform the generated code in a contorted way so that the
lines remain in order.

Comments also in http://lua-users.org/lists/lua-l/2009-05/msg00300.html .

On Wed, Jun 16, 2010 at 12:04 PM, steve donovan
<steve.j.donovan@gmail.com> wrote:
> That's the basic problem - there are no guarantees that chaining
> arbitrary filters will work, and the results would often be
> order-dependent. It would very much be a 'caveat emptor' situation.

Without altering the token filter API, we could define a single token
filter--let's call it a "meta-filter"--that permits multiple clients
to plug into the meta-filter if they follow certain rules.  As a token
filter, you can have only one meta-filter active at a time, but the
meta-filter can itself define a mechanism to chain rules from multiple
clients.

On Wed, Jun 16, 2010 at 11:50 AM, Jerome Vuarand
<jerome.vuarand@gmail.com> wrote:
> While saving bytecode reduces portability, what prevents you from
> re-serializing the output of the token filter in source form ? Are
> there token that could be emited by a token filter, but couldn't go
> through the unmodified Lua lexer ?

There are a couple things that can be expressed in bytecode but not
source.  Direct access to debugging information is one, although code
transformations like Lhf mentioned can help some (and Lua syntax could
be extended to support directly injecting debug info).  Another
difficulty is if you have a macro containing statements and that macro
expands inside of an expression.  Lua syntax doesn't support
statements inside expressions without the overhead of an anonymous
function construction.  Metalua overcomes this by emitting bytecode.
It might not be necessary though.  Although not the easiest to do, my
source optimizer [1] inlines statements-in-expressions by translating
the code into regular expression-in-statement form, and I've thought
this could be useful in Metalua to output source, especially given
that LuaJIT2 isn't compatible with Lua 5.1 bytecode.

[1] http://lua-users.org/wiki/SourceOptimizer