Re: [ANN] LPeg 0.11

[Patrick Donnelly <batrick@...>]

On Mon, Apr 1, 2013 at 6:21 PM, Miles Bader <miles@gnu.org> wrote:
> Roberto Ierusalimschy <roberto@inf.puc-rio.br> writes:
>> I am not sure I understood what you want. If you do not want to keep
>> any captures from ONE_STATEMENT, why does it have captures in the
>> first place?
>
> Any results from ONE_STATEMENT are already recorded via side-effects,
> so any captures inside it have already been used, and I want to
> discard them for the "next iteration" (of ONE_STATEMENT).

LPeg can't know those captures can be discarded and so can't "flush"
the captures. The reason for this is quite simple and you brought it
up: side-effects. Consider (automagically) flushing a pattern's
captures which calls capture functions with side-effects. However, the
enclosing pattern *does not match* and so those side-effects should
never have happened.

> Trying this in real life leads to memory explosion, whereas just using
> an explicit loop outside of LPeg doesn't, so clearly something's
> being kept around inside LPeg between iterations.

You talk about memory explosion, do you mean LPeg stack overflow? If
so, the solution I've used for an LPeg Lua source formatter is to
flatten large numbers of captures at strategic points into a single
capture using:

-- The formatter uses captures to indent code. We necessarily use thousands and
-- thousands of them. At various strategic points, we concatenate these captures
-- so we don't overflow the Lua stack.
local function FLATTEN (pattern)
  return Ct(pattern) / concat;
end


The other solution is to use match-time captures which flush captures
-- which you wanted in your referenced posting. Unfortunately
match-time captures slow down matching considerably.

--
- Patrick Donnelly