RE: Pluto: heavy-duty persistence for Lua

["Virgil Smith" <Virgil@...>]

Alas someone has beat me to it. :-)

I wrote such a system (incompletely) for Lua v4.  That was part of a
commercial non-open source project, and I did it simply because it was
easier than implementing any other data format/file system for the project.

Immediately after that Lua5 was released with features that would make
reworking my "LuaDat" system (name internally used only and I don't know if
the it is already taken) quite aesthetically pleasing.  However, I've never
quite gotten the time to go back and recode it (though I do have
documentation for a stream format and programming interface).

---
>From my previous efforts/considerations I have the following comments.

1. Your use of chunk readers/writers is a very nice orthogonality touch.

2. If closures hold you back on support for Heavy Userdata, you could
instead rely on tables.  This would mean that a userdata's serialization
metamethod should return a table rather than a closure.  This table can then
be persisted in the userdata's place.  A likely method for the
deserialization would be for this "stand in" table to include a
deserialization metamethod.  This is/was my current design, but I like your
closure approach better.

3. Tables should be checked for a serialization metamethod just like
userdata.  The reason for this is that table's are Lua side "objects" or
"type extensions" and may therefore need extended handling compared to a
"primitive" type.  Examples of such "needs" would be for proper handling of
"cached" values after deserialization to account for changes in the
environment, simply tossing out those cached values to save space in the
stream, reloading/linking of libraries, etc.

4. Closures can also represent an "extended type" in Lua, but I have
absolutely no clue on how they should be given "special consideration" in
the way that heavy userdata and tables should.  Others who frequent the list
can probably make useful comments on this issue.  However, I'd recommend
keeping this in mind, but not stalling anything because of it.  Why?  Just
because I personally wouldn't know what to do with it and therefore I
believe most people wouldn't either. (and yes I am the center of my own
universe :-)

5. In my own serialization of heavy userdata I found it necessary to add my
own "special type" called a blob (aren't I so original).  My blob's were
very much like Lua strings in that they had a length and could store any
binary data, but they weren't allocated through Lua, they did NOT have Lua's
string value comparison semantics, and they did NOT require performing a
copy operation to place them on the Lua stack.  Basically this amounts to an
arbitrary block of data in the stream, that a userdata can use to store its
internal data in whatever format it wishes (in my case this included a lot
of large pictures saved in the format used by a particular camera SDK).  A
userdata could include one of these in its "stand-in" table to be
serialized.

This need may be well served by an existing buffering library, but I've not
investigated that.  Also, a buffer is probably not the best way to handle
this anyway.  Probably the best way is through writer/reader functions so
that this "non-Lua" data could be serialized piece by piece if appropriate
(as opposed to allocating buffers to hold all of the data needed by a
particular userdata object).  This was the only part of my design that I was
not happy with, so I'll give this some more thought and let you know what I
come up with (hopefully you'll actually like it :-)

---
Well that's more than I really had time for now, but I'll check out your
code later.

Thanks for taking the time to put this forth!




-----Original Message-----
From: lua-bounces@bazar2.conectiva.com.br
[mailto:lua-bounces@bazar2.conectiva.com.br]On Behalf Of benjamin
sunshine-hill
Sent: Wednesday, June 02, 2004 5:03 PM
To: lua@bazar2.conectiva.com.br
Subject: Pluto: heavy-duty persistence for Lua


I've just finished fixing up the first alpha version of Pluto, a library
that allows persistence of arbitrary lua objects. Its prime features are its
ability to store functions and threads, and its support for "permanent
objects". A wiki page about it is at http://lua-users.org/wiki/PlutoLibrary
. Feature requests and bug reports are requested.

Ben