load() and standard iterators for reading files in 5.2

[Duncan Cross <duncan.cross@...>]

On Thu, Feb 25, 2010 at 8:52 PM, Roberto Ierusalimschy
<roberto@inf.puc-rio.br> wrote:
>> not entirely convinced that the only reason you would ever want to use
>> a "loadfilein" is sandboxing, but then in truth I don't have an actual
>> counterexample to support that. Sorry for the noise, anyway.)
>
> Again maybe I was not very clear. The specificity of the situation was
> not only the sandboxing, but the absence of the io lib. With the io
> lib, you should be able to do a loadfilein like here:
>
>  loadin(env, io.lines(filename))
>
> (except that this does not work, because 'io.lines' cut the newlines
> from the file; we intend to fix that somehow...)
>
> -- Roberto
>

Oh, that's a good point. I had assumed that you would need to do
something pretty involved with coroutines etc. to write your own
equivalent function, which might be a bit of a tall order for the
less-experienced - I hadn't thought about using io.lines() as a
standard iterator for load(), that is quite a powerful idea.

There's a few solutions I can think of to the newlines problem:
+ Add a new extra boolean parameter to io.lines() to specify that you
want to preserve them
+ Allow functions passed to load() to return multiple strings instead
of just the one, and change io.lines() iterators to yield two results
each time, the line and the new line character(s)
+ Add a new function, a generalised form of io.lines(), something like
io.reader(filename, format) where 'format' is the same kind of thing
as the argument(s) passed to file:read(). If you pass "*l", the
iterator returned is identical to one created by io.lines() (and would
have the same problem); you could instead pass a number or "*a", which
would create an iterator that would not strip the newlines.

Personally I like the third option, as it seems like a more general
solution. Any which way, I think it is worth finding a good solution
in time for the final release of 5.2.

-Duncan