Re: lex patch

[Edgar Toernig <froese@...>]

Philippe Lhoste wrote:
> 
> > - EOLs in long strings and escaped EOLs in strings are converted
> > to a single "\n"
> 
> ? I am not sure to understand fully.

x = [[
some
lines
of
text]]

and

x = "foo\
bar"

With current Lua you don't know what is in x afterwards.  It depends
on the EOLs in the source and what the stdio library does with them.
With the patch you always get a "\n" for the embedded EOLs.


> > - Has its own ctype macros to make identifiers locale independent.
> 
> Well, originally, identifiers *are* voluntarily locale dependent, at least
> it is documented... But as Visual Basic shown, if it is friendly for a locale
> developper (which can type identifiers in its own language), it is
> problematic for exchanging source code over nations.

Not only over nations.  Using locales is a user choice and some people
(like me) always use the C locale and will get a lot of problems with
code that uses "strange" identifiers.  Even trying to figure out what
locale the author used may become difficult.  Converting these identifier
may be impossible.  Switching to "his" locale isn't easy either.  So
IMHO it's best to define a well defined subset of characters to be
used for identifiers ([a-zA-Z_0-9]) and the problem will never occur.[1]

> > - Adds "long comments": --[[ comment ]] (see Lua-Wiki power patches).
            ^^^^ [that should have been "block comments" - ET]
> 
> I believe Lua authors have already implemented it (if not released).

On the Lua-users Wiki someone mentioned that they are to be consider for
inclusion.  But the work4 snapshot didn't include them so I thought it's
a good time to remind them ;-)

> I want to implement its lexing in SciTE.

If you already have a long-string parser in SciTE you shouldn't have a
problem adding them - the block comments use the same parser.


> > - Changes the "\123" escape from decimal to octal as it is done in
> > _all_ other languages I know.
> 
> Mmm, I don't like it.

I expected that ;-)

> Lua don't have to be like "all other languages".

But Lua tries to be like "all other languages" regarding string parsing.
They implemented all the commonly used sequences (\n\t\a\f\r).  There's
only one exception: \123.  In all other implementations the digits are
considered an octal number and Lua takes is as a decimal one. That's
asking for problems.

> Octal is an old survivance of the PDP days, and isn't used much outside this
> notation. Most programmers are unconfortable with it (wild guess :-).

Yes, these octal numbers are not nice.  But even worse is that Lua changes
only that part of a well established concept.  If they'd chosen a different
escape mechanism (i.e. "~n" instead of "\n") I would've never complained...

> Of course, it is confusing (I remember Luiz having made the confusion in a
> line of the Lua 4.0 code :-) but it is clear and easy to use.

Well, even the ones who should know best run into this pit.  What more
to say? ;-)

> And more annoyingly, this patch breaks backward compatibility.

A tool that checks for these sequences and issues a warning shouldn't
be that difficult...  (could warn about block comments and other
incompatibilities like new reserved words etc, too)

Ciao, ET.


[1] Btw, while identifiers respect your locale settings, numbers do not
(0.1 vs 0,1).