new library for lexical analysis

[Luiz Henrique de Figueiredo <lhf@...>]

I've written a Lua library for lexical analysis of Lua code:

	http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/ltokens.tar.gz

This library is based directly on the Lua lexer and so is guaranteed to be
consistent with what the Lua parser sees. (I wish a similar reuse of the
Lua parser was also easy to write.)

The package also contains a simple utility for compressing Lua source code
by removing comments and superfluous whitespace.

Unlike other libraries, this one uses the internal core functions, not just
the official C API. On the other hand, like luac.c, it does so politely.
Just be warned that the internal API may change without notice.

Enjoy.
--lhf