Re: Problems with lpeg.Cmt captures

[Glenn McAllister <glenn@...>]

Roberto Ierusalimschy wrote:
> > So for some reason, despite returning the fact that I said the match 
> > succeeded, it appears to have failed and is calling it again.  At this 
> > point, I'm not sure what captured element is being provided, unless its the 
> > entire table of chunks.
>
> In your trace there is this sequence, just after the Cmt execution:
>
> || s: | three| stck: 12 c: 16  89: commit -> 102
> || s: | three| stck: 11 c: 16  102: ret
> || s: | three| stck: 10 c: 16  197: failtwice
>
> The commit seems consistent with the choice in your rule (" + Cs...");
> The return seems to finish the rule EndifExpr. But then there is this
> failtwice. This instruction is generated only by a not predicate ('a-b'
> or '-b'). Are you sure your pattern does not have this predicate?
>
> -- Roberto

I probably should have started this thread stating out front that I'm 
porting the Java StringTemplate template library/engine to Lua.  It's 
the StringTemplate spec that is determining what I need to do. 
Specifically regarding whitespace, you can see what I'm working from 
here: http://tinyurl.com/65l87h (the 'Whitespace in Conditionals' 
section at the bottom of the page).

I think this line is the culprit:

Expr = -EndifExpr * (EscapeExpr + CommentExpr + IfExpr + TemplateRefExpr 
+ AttrRefExpr)

I was finding that if I didn't put the -EndifExpr then $endif$ was being 
treated as an attribute reference (AttrRefExpr) that effectively 
translates into an empty string.  When I just have

Expr = EscapeExpr + CommentExpr + IfExpr + TemplateRefExpr + AttrRefExpr

I get a bunch of failures where the IfExpr isn't getting parsed correctly.

Here's the entire grammar so far.  I'm new to PEGs as I've previously 
pointed out, so I'm probably doing something wrong; any advice would be 
appreciated.  Sorry if this wraps badly for someone.

-- Grammar terminals
local s = {
    N = R'09',
    AZ = R('__','az','AZ','\127\255'),
    NEWLINE = S'\n\r',
    SPACE = S' \t',
    WS = S' \t\n\r',
    SEMI = S';',
    COMMA = S',',
    PERIOD = S'.',
    SEPARATOR = P'separator',
    EQUALS = P'=',
    DQUOTE = S'"',
    NULL = P'null',
    EPSILON = P(true),
    ESCAPE = S'\\',
    BANG = P'!',
    EXPR_START_DOLLAR = P'$' - S'\\',
    EXPR_END_DOLLAR = P'$',
    EXPR_START_BRACKET = P'<' - S'\\',
    EXPR_END_BRACKET = P'>',
    LBRACE = P'(',
    RBRACE = P')',
    IF = P'if',
    ENDIF = P'endif'
}

-- A bunch of V'foo' assignments happen around here so the grammer
-- looks a little cleaner (from my pov).

local grammar = {
    "Template",

    Template = TemplateBody * -1,

    TemplateBody = Ct(Chunk^1),

    Chunk = Newline + Literal + Expr,

    Expr = -EndifExpr * (EscapeExpr + CommentExpr + IfExpr + 
TemplateRefExpr + AttrRefExpr),
    --Expr = IfExpr + EscapeExpr + CommentExpr + TemplateRefExpr + 
AttrRefExpr,

    Newline = C(s.NEWLINE) / newNewline,

    Literal = Cs(((LiteralEscape + 1) - (ExprStart + Newline))^1) / 
newLiteral,

    LiteralEscape = (s.ESCAPE * S'$<') / literalEscapes,

    AttrRef =  C((1 - (ExprEnd + s.SEMI + s.PERIOD))^1),

    AttrProp = (s.PERIOD * C((1 - (ExprEnd + s.SEMI))^1)) +
               C(s.EPSILON),

    AttrOpts = (s.SEMI * s.SPACE^0 *
                    Ct(AttrOpt * (s.COMMA * s.SPACE^0 * AttrOpt)^0)) +
               C(s.EPSILON),  -- ensures we always get something for 
the options

    AttrOpt = Ct(C((1 - (s.EQUALS + s.COMMA))^1) * s.EQUALS * s.DQUOTE *
                Cs(((((s.ESCAPE * S'ntr ')/exprEscapes) + 1) - 
s.DQUOTE)^0)
                * s.DQUOTE) +
              Ct(C((1 - (s.COMMA + s.SPACE + ExprEnd))^1) * C(s.EPSILON)),

    AttrRefExpr = ExprStart *
                    (AttrRef * AttrProp * AttrOpts / newAttrRefExpr) *
                    ExprEnd,

    CommentExpr = ExprStart * s.BANG * (1 - s.BANG)^0 * s.BANG * ExprEnd,

    EscapeExpr = ExprStart *
                    Ct(Cs(((s.ESCAPE * S'ntr ') / exprEscapes))^1) / 
newEscapeExpr *
                    ExprEnd,

    TemplateRef = C((1 - s.LBRACE)^1),

    TemplateParamList = TemplateParam * (s.WS^0 * s.COMMA * s.WS^0 * 
TemplateParam)^0,

    TemplateParam = Name * s.WS^0 * s.EQUALS * s.WS^0 * Name,

    Name = C(s.AZ * (s.AZ + s.N)^0),

    TemplateRefExpr = ExprStart * (TemplateRef *
                        s.LBRACE * s.WS^0 *
                        (Ct(TemplateParamList) + s.EPSILON) * s.WS^0 *
                        s.RBRACE) / newTemplateRef *
                        ExprEnd,

    EndifExpr = Cmt(Cb(1) * ExprStart * s.ENDIF * ExprEnd * s.NEWLINE,
                    function(s,i,a)
                        if a.isA then
                            if a:isA(NewlineChunk) then
                                return i, "kill"
                            else
                                return false
                            end
                        else
                            return false
                        end
                    end) +
                Cs((ExprStart * C(s.ENDIF) * ExprEnd) / "dontkill"),

    -- Because we are creating an anonymous embedded template, we need to
    -- pass in options (scanner and auto_indent) that the template 
cares about
    IfExpr = ExprStart * s.IF * s.LBRACE * IfExprAttr * IfExprProp * 
s.RBRACE * ExprEnd *
                --s.NEWLINE^0 * C((1 - (ExprStart * s.ENDIF))^0) * 
s.NEWLINE^0 *
                s.NEWLINE^0 * Ct(Chunk^1) *
                EndifExpr *
                Carg(1) * Carg(2) / newIf,

    IfExprAttr = C((1 - (s.PERIOD + s.RBRACE + ExprEnd))^1),

    -- This one is tricky, as we need to deal with indirect property
    -- references, which are ofset by braces, which also happen to 
delimit the
    -- attribute/property of the if expression.
    IfExprProp = (s.PERIOD * C(s.LBRACE * (1 - (ExprEnd + s.RBRACE))^1 
* s.RBRACE)) +
                 (s.PERIOD * C((1 - (ExprEnd + s.RBRACE))^1)) +
                 C(s.EPSILON),
}

--
Glenn McAllister     <glenn@somanetworks.com>      +1 416 348 1594
SOMA Networks, Inc.  http://www.somanetworks.com/  +1 416 977 1414