Re: ANN: Serpent: Lua serializer and pretty printer

[Paul K <paulclinger@...>]

Hi All,

I released a new Serpent version that incorporates the options we've
been discussing earlier. It's mostly as described in my previous
email, but I also added nohuge option (for those who are on a platform
that generates proper strings in those cases, but want a bit faster
implementation) and changed "long" to "block" method. As the interface
has changed, I also updated the tests, the documentation, and the
benchmark.

>>>> (2) A "--[[table: 0x9aa7988]]" style comment is appended after each
>>>> (3) The keys are not currently being sorted.
>>>> (10) Dumps of bytecode are probably not useful for pretty printing
>>>> (12) In "nil values are included when expected ({1, nil, 3} instead of

All these items should be addressed: block(a, {comment=false,
sortkeys=true, nocode=true, sparse=true}), should produce diffable
output.

Custom formatters also allow making tweaks to the output, for example,
printing of ASTs in Metalua format. You can use the following code:

print((require "serpent").block(ast, {comment = false, custom =
  function(tag,head,body,tail)
    local out = head..body..tail
    if tag:find('^lineinfo') then
      out = out:gsub("\n%s+", "") -- collapse lineinfo to one line
    elseif tag == '' then
      body = body:gsub('%s*lineinfo = [^\n]+', '')
      local _,_,atag = body:find('tag = "(%w+)"%s*$')
      if atag then
        out = "`"..atag..head.. body:gsub('%s*tag = "%w+"%s*$', '')..tail
        out = out:gsub("\n%s+", ""):gsub(",}","}")
      else out = head..body..tail end
    end
    return tag..out
  end}))

to generate this output:

{
  `Call{`Id{"require"},`String{"foobar"}},
  `Local{{`Id{"foo"}},{`Id{"foo"}}},
  `Local{{`Id{"y"}},{`Number{5}}},
  `Do{`Local{{`Id{"y"}},{`Number{6}}},`Local{{`Id{"y"}},{`Number{7}}}},
  `Localrec{{`Id{"test"}},{`Function{{`Id{"x"}},{`Call{`Id{"print"},`String{"123"},`Id{"x"},`Id{"y"},`Id{"z"}}}}}},
  `Call{`Id{"bar"},`Number{123}},
  `Set{{`Id{"bar"}},{`Function{{},{}}}},
  `Local{{`Id{"g"}},{`Function{{`Id{"w"}},{`Return{`Op{"mul",`Id{"w"},`Number{2}}}}}}},
  `Set{{`Id{"g"}},{`Number{1}}},
  `Call{`Id{"g"},`Number{1}},
}

The updated version is on github: https://github.com/pkulchenko/serpent.

Paul.

On Tue, Jun 12, 2012 at 12:33 PM, Paul K <paulclinger@yahoo.com> wrote:
> Hi David,
>
> Here is what I came up with based on this discussion:
>
> Methods:
>
>  dump -- dump fully serialized value
>  line -- pretty single-line output, no self-ref section
>  long -- pretty multi-line output, no self-ref section
>
> Options:
>
>  name     nil/string -- name; triggers full serialization
>  indent   nil/string -- indentation; triggers long multi-line output
>  comment  True/false -- provide stringified value in a comment
>  sortkeys true/False -- sort keys using alphanum sorting
>  sparse   true/False -- force sparse encoding (no nil filling based on #t)
>  compact  true/False -- remove spaces around = and after ,
>  fatal    true/False -- raise fatal error on non-serilizable values
>  custom   function   -- to provide custom output for tables
>  nocode   true/False -- disables bytecode serialization for easy comparison
>
> Usage:
>
> dump(a, ...) serialize(a, {name = '_', compact = true, comment =
> false, sparse = true ...})
> line(a, ...) serialize(a, {sortkeys = true, ...})
> long(a, ...) serialize(a, {indent = '  ', sortkeys = true, ...})
>
> You can also provide parameters that overwrite defaults. For example,
> for diff optimized output:
>
> diff(a) dump(a, {nocode = true, indent = ' '})
>
> Or for AST printing:
>
> asast(a, ...) serialize(a, {custom = function(tag, head, body, tail), ...})
>
> The custom function can print lineinfo on one line or put the tag
> value first and so on.
>
> I also plan to use the alphanumeric sort as you suggested
> (http://notebook.kulchenko.com/algorithms/alphanumeric-natural-sorting-for-humans-in-lua),
> but only when the sortkeys option is on.
>
> Paul.
>
> On Tue, Jun 12, 2012 at 12:00 AM, David Manura <dm.lua@math2.org> wrote:
>> On Sun, Jun 10, 2012 at 12:38 AM, Paul K <paulclinger@yahoo.com> wrote:
>>>> (2) A "--[[table: 0x9aa7988]]" style comment is appended after each
>>>> (3) The keys are not currently being sorted.
>>>> (10) Dumps of bytecode are probably not useful for pretty printing
>>>> (12) In "nil values are included when expected ({1, nil, 3} instead of
>>
>> The above four points all are related to whether the dumper should
>> have the property of deterministic output under structural equality.
>> Either way, this property may be worth discussing in Features.
>>
>> Arguments for deterministic output:
>>
>> - Dumping may be used as a poor man's test of structural equality:
>> serialize(a) == serialize(b).  This can be useful in test suites.
>> More sophisticated alternatives include
>> https://github.com/silentbicycle/tamale .
>>
>> - Sizes of diffs between dumps is reduced.  This may be useful when
>> the serializer is used to write data or configuration files to be
>> maintained under revision control.  It may also be useful in
>> debugging, such as diffing dumps from two separate runs.  However,
>> deterministic output is not the only factor affecting diff size.
>>
>> - Sorting may improve readability, particularly for tables with many
>> elements (e.g. thousands).  However, optimal sort order is subjective
>> -- http://www.codinghorror.com/blog/2007/12/sorting-for-humans-natural-sort-order.html
>> .
>>
>> Arguments against:
>>
>> - Impact on performance and code complexity?
>>
>> - Not needed in some cases.
>>