RE: Syntax trees, macros, and reflexive Lua evolution

["Nick Trout" <nick@...>]

> On Behalf Of Dr. Rich Artym
> The never-ending suggestions for surface syntax improvements and Mark
> Hamburg's mention back in September of Lua macros and his reference to
> that great essay at

I was thinking of adding a Lua library with hooks in the compiler, like
debug hooks, so that you could alter the behaviour of the parser and
byte code generation. Using this you potentially add new syntax for say,
const tables (like enums), or static checking for functions. E.g. you
may have list of functions you have exported from your app or other
scripts and you could pick up linkage errors at compile time. 

I feel where Lua needs to go now is towards adding more support for
static language type features. All these minor syntax changes aren't
advancing the language, and most of them can be done in the language
already, either functionally or with minor patches. My choice has been
never to patch the Lua core, as essentially you create your own language
fork. The authors have usually come up with better configuration scripts
for the code base or language features which allow customisation of the
environment. 

Unfortunately I have no time to do this and instead have written a
script pre-processor which checks luac disassembly for function calls to
do some static checking and substitute constants. This could possibly be
done by having a lparselib.c file like ldblib.c/ldebug.c which allow
hooks to query and change the behaviour of the parser. Possibly you'd
have to be able to add "hints" to the code to give clues about the
context of your changes. E.g.:

	-- const enum type
	colours = [enum] {		-- note the hint
		red = 0xff0000,
		blue = 0x00ff00,
		-- etc --
	}

In this instance, your Lua config might choose to save the colour
enumerations and then substitute the values at compile time when it
meets colours.red again. This saves table lookups and generates less
code. Of course there are other solutions to this (locals etc) but I'm
using this as an example.

This might be done with pragmas, which would be specific to the user
config, and not part of the language. For instance your particular
config might add a pragma to do static checking of function calls

#pragma fn_static

		foo(1, 2, "buckle", my, shoe)

This might be implemented by picking up a bad keyword "#pragma",
recognising it as one your keywords and changing the decision of the
compiler to flag this, kicking in another compilation context.

The hybrid compilation process would be: 

- apply parsing hooks in parse lua_State
- compile script into user lua_State
	- whilst parsing hooks pass data to parse lua_State
	- parse callbacks optionally alter compilation
- code generated in user lua_State

You wouldn't be able to put "parse hooks" into the parse hook lua_State
as you'd probably get infinite recursion. This is somewhere between a
pre-processor and a compiler.

In addition, if you did want to have Lua macros, you could, by possibly
to this by adding a pragma/commmand to generate code, running this in
the parse lua_State and returning the result to be compiled in the user
lua_State. E.g.

-- in user script
#macro_define \
	squares (x)	\
		local t = "{"
		for i = 1,x do t=t.."{"..i..","..i*i.."}," end \
		return t.."}" \
	end

-- usage of macro:
-- returns a table string containing map of squares which
-- gets stuffed into the compiler in place of the #macro symbol
#macro squares(10)

I don't really see macros as a big part of Lua, this is just an example
of what might be possible with a hybrid pre-processor/compiler. I've
probably greatly over simplified problems involved in this and if I had
more time perhaps I could demonstrate a proof of concept.

The other thing is that the VM would have to be able to work without
this as Lua can currently be run. This is pretty important to small
memory systems and the compilation stage will generate significant
memory demands.

Nick