Token filters, require() and lua's -i switch

[Thomas Lauer <thomas.lauer@...>]

FWIW and further to some posts of last week, here's an adapted version
of Luiz' token filter code, Lua side. The C side is unchanged, although
I think that it might be worthwhile to explore the possibility of
changing the C code such that it doesn't call one function to do two
separate things.

Anyway, this is my current filter, stripped of all actual filter code
(i.e. the do(This|That)Token calls are placeholders).

-- snip
local pre_filter
local function filter(get,put)
	put()
	while true do
		local line1,token1,value1=get()
		if token1=="thistoken" then doThisToken(line1,token1,value1)
		elseif token1=="thattoken" then doThatToken(line1,token1,value1)
		elseif token1=="<eof>" then -- found eof, switch back to pre_filter
			FILTER=pre_filter
			return line1,token1,value1
		else put(line1,token1,value1) end
	end
end

pre_filter=function(get,source)
	FILTER=coroutine.wrap(filter)
	FILTER(get,coroutine.yield)
end

FILTER=pre_filter
-- snap

So far, this seems to work with all sorts of require()'s, loadstring()'s
etc. I keep throwing at it, so long as an actual <eof> token is seen
(which, as Luiz pointed out, may not arrive if the source contains an
error... this is no big problem for file input (if there's an error, all
bets are off anyway), but it does mean that interactive use (along the
lines of 'lua -i -l tokenf test.lua') will stop working after an error
occurred).

Here's another oddity with -i. When entering interactive mode, the
session executes exactly one '=' command. The next '=' command does
nothing and then Lua just prints "cannot resume dead coroutine".

Fortunately, this is relatively easy to fix: a '=X' is translated into a
'return X' (see lua.c's pushline() function). Instead of doing a 'return
X' just do a 'do return X end':

lua_pushfstring(L, "do return %s end", b+1); /* change it to `return' */

-- 
cheers  thomasl

web : http://thomaslauer.com/start