Re: What's up with token filters (Re: New operators?)

[Thomas Lauer <thomas.lauer@...>]

Rici Lake <lua@ricilake.net> and...
> David Given wrote:
> > It occurs to me that a reasonably elegant way of implementing non-trivial
> > syntax modifications to Lua would be to have a totally separate, off-line
> > compiler --- most likely written in Lua --- that reads in a Lua chunk,
> > transforms it somehow, and then outputs it again as vanilla Lua source that
> > can be passed to the real interpreter.

I have toyed with that idea as well, using Lua as a sort of backend for
an extended version of itself. The biggest hurdle I faced was to make
sure that the "pre-compiler" (which was admittedly a rather simple
affair) spits out 100% correct Lua code. Otherwise the diagnostics are
hell.

Also I have not hit upon a clear, standardised mechanism (other than
pipes) to interface between the pre-compiling phase and the Lua backend.

> I don't think that would be adequate. There are a number of useful and
> interesting constructions which can be implemented by generating VM
> code, but cannot be implemented (efficiently) as source code transforms.

If "source code transforms" means a thin layer of search-and-replace,
you're probably right. But a fully-fledged AST compiler might do a very
good job of optimising things. And even if some features are not
compiled as efficiently as Lua could do them, other parts might be much
more efficient, exactly because an AST-based compiler can do things that
the fast Lua compiler can't.

Still, an officially sanctioned VM assembler would have some appeal.