Re: reading big files

[Roberto Ierusalimschy <roberto@...>]

> I tried to see what is happening by looking into liolib.c (version
> 5.2.2). However, I am not sure if I understood the code well
> enough. Therefore I am asking if someone more knowledgeable will
> correct me where I err.
> 
> The impression I got is that stream:read(buffersize) is reading in a
> buffer a chunk of size LUAL_BUFFERSIZE, and then at each step of fread
> in function read_all enlarges the size (up to a certain maximum). In
> the process luaL_prepbuffsize is called and the data are transferred
> with a memcpy to a new and larger memory chunk. Do I see correctly
> that in the successive steps while reading very large chunks, this
> leads to memcpy the data over and over until everything is read?

This is what Lua 5.1 did. Lua 5.2 preallocates a buffer with the given
size and reads the whole chunk at once. The relevant function is
'read_chars', not 'read_all'.

('read_all' is used only when you read the entire file at once, with
f:read("a"). Even then, it doubles the size of the buffer at each step,
so the number of copies is log(n).)

-- Roberto