Re: reading big files

[Luiz Henrique de Figueiredo <lhf@...>]

> In the processing stage I am taking out parts of about 200 bytes a time out of a larger buffer (buffersize is set to about 10 MB).

Large buffers probably do no good. You probably want to avoid the
concatenations done in the io library for large buffers (though it
is done a log number of times). So, stick to a buffer size equal to
LUAL_BUFFERSIZE, which by default is the system BUFSIZ.

The test program below gives this result.

B	time
1048576	0.08
524288	0.09
262144	0.04
131072	0.06
65536	0.06
32768	0.07
16384	0.11
8192	0.08
4096	0.08
2048	0.11
1024	0.13
512	0.18
256	0.27
128	0.46
64	0.83
32	1.97
16	3.65
8	6.71

Note how 4096, which is probably BUFSIZ here, gives the same performance
as 1048576, though in this case it seems that 262144 is the best. Once
you starting adding processing, these numbers will change. So, I suggest
you stick to reading 200 bytes at a time, which is what your program
needs.

-- test program

F="626Mfile"
local B=1024
B=B*1024
while B>=1 do
 assert(io.input(F))
 local t=os.clock()
 while true do
  if io.read(B)==nil then break end
 end
 d=os.clock()-t
 print(B,d)
 B=B/2
 collectgarbage()
end