Re: Luarc - a continuation-based web server

["Mark Meijer" <meijer78@...>]

2008/9/15 Fabio Mascarenhas <mascarenhas@acm.org>:
> On Mon, Sep 15, 2008 at 3:04 AM, Sacado Sacado <sacado.sacado@gmail.com> wrote:
>> The real big problem I can see as for now is memory use... Since the Lua
>> interpreter is so small, I really can see it grow, kb after kb, every time a
>> request is thrown.. This is frightening... :) But, even after tens of
>> thousands of requests, the server uses less than 4MB. In nowadays'
>> standards, this is nothing...
>
> Implementing expiration of continuations (or sending them to the
> client, see below) takes care of the memory issue, no? Unless you've
> got a leak on your server. :-)

Many on this list will already know the following, but I didn't when I
ran into it many months ago. And that is the fact that tables keep
internal nodes which are used to hold the table entries, even for
entries that have since been reset to nil. At least, it keeps them
until a table resize is triggered, but in the general case it is not
practically predictable when this happens. The reason it keeps these
"dummy" nodes for entries that no longer exist, is sothat entries can
be removed from a table while iterating over it. Usually, tables are
small enough, or table resizes are triggered often enough, for this
not to be a significant problem.

Therefore, I really don't know if this is the problem you are seeing.
We're not talking about a lot of extra memory potentially being held
per table here (don't remember the size in bytes of a single node, but
it's something like 16 or 32 bytes or thereabouts). So if you're
somehow "leaking" multiple megabytes, then it could only be attributed
to the effect described above, if your tables are mostly indexed with
the same set of keys (which means a table resize won't get triggered),
and then only if you have lots of those tables (or few of those tables
with a very large but fairly constant set of keys).

So, I think it's unlikely you are facing this problem, actually. But
it's probably the only use of memory by Lua itself that you can't
directly account for by looking at your Lua code. So to be sure, I
just thought I'd mention it. This memory use also isn't reported by
querying the garbage collector from a Lua script. You can only tell
from looking at the size of the OS process in question (AFAIK).

Btw. I hope I'm explaining this all correctly. I only know what others
on this list have told me about it all those months ago, and I might
have misunderstood or remembered some things wrong.