Re: lpeg and streams

[Thomas Harning <harningt@...>]

On Aug 18, 2008, at 4:15 PM, Scott Fial wrote:
> I am new to lpeg (and lua).  Lpeg seems to require that the entire  
> input string be available prior to parsing.  I would like to be able  
> to use lpeg to parse a stream of incoming data.  I would also like  
> to be able to use it for parsing huge files.  In both cases, I won't  
> have the entire input in memory.  I need the parser to be able to  
> accept more data as it arrives, while maintaining its state.
>
> Ideally, I would like lpeg.match() to accept an optional function  
> parameter that the parser will call when it hits the end of the  
> current input string and is ready for more input.  Then I can pass  
> it a function to read the next few megs from a file, or call  
> coroutine.yield until there is more data available on the  
> communications channel.
>
> Is there already an elegant *lua* way to handle the situations I  
> have described?  If not, would the callback architecture I've  
> suggested someday be possible for lpeg?
The way LPEG works (and most parsers in general), it seems that all  
old data must be available, though it would seem LPEG 'could' work in  
a way that permitted more data to be fed in later... still requiring  
that the old data be maintained.

One potential mechanism could be to permit passing a memory-mapped  
file to LPEG, pending you can map the entire file...  This would  
require LPEG to accept a userdata object and collect the memory  
pointer and length to the file data.