Re: load() and standard iterators for reading files in 5.2

[Mark Hamburg <mark@...>]

On Feb 26, 2010, at 5:29 AM, Roberto Ierusalimschy wrote:

>> There's a few solutions I can think of to the newlines problem:
>> + Add a new extra boolean parameter to io.lines() to specify that you
>> want to preserve them
>> + Allow functions passed to load() to return multiple strings instead
>> of just the one, and change io.lines() iterators to yield two results
>> each time, the line and the new line character(s)
>> + Add a new function, a generalised form of io.lines(), something like
>> io.reader(filename, format) where 'format' is the same kind of thing
>> as the argument(s) passed to file:read(). If you pass "*l", the
>> iterator returned is identical to one created by io.lines() (and would
>> have the same problem); you could instead pass a number or "*a", which
>> would create an iterator that would not strip the newlines.
>> 
>> Personally I like the third option, as it seems like a more general
>> solution. Any which way, I think it is worth finding a good solution
>> in time for the final release of 5.2.
> 
> We implemented the first option, but currently we are inclined to the
> third one. As Steve pointed out, it would have other uses too.

The nice thing about the third option is that the notion of "reader" is already present in the APIs. There just hasn't been much support for creating one.

Speaking of which, could one implement a slow version of token filters by having an option to run the reader output through a lexer and then processing the material that came out of that? Using the sequence library notion from a couple months ago:

	seq( io.reader( "foo.lua", "*l" ) )( syntax.lexer )( my_token_filter_1 )( my_token_filter_2 )

Mark