Re: Understanding LTN12 filters and pumps

["Pierre Chapuis" <catwell@...>]

> I've been reading about LTN12:
> http://w3.impa.br/~diego/software/luasocket/ltn12.html
> it looks very useful, but I'm having trouble understanding just how it
> works.
>
> Am I correct in that a filter is expected to accept a string, and return a
> (probably different) string? And a pump is just "read data from source and
> write it to sink"?
>
> In toying around with this, I implemented a "tobase64" function. It takes
> a
> source and a sink, and keeps reading data from the source until no more is
> available, converting it to base64, and writing it to the sink. It seems
> like a simple enough system, able to process large amounts of data without
> using a lot of memory, and able to be chained (have the source or sink be
> another such function). But I think this isn't how filters normally work?
> What's the purpose of separating filter and pump?

If you really want to understand you should
read http://lua-users.org/wiki/FiltersSourcesAndSinks

I will just try to sum it up. ltn12 works on *streams*
of data that flows through the system. Data comes from
a source, goes through filters and ends up in sinks,
so the system looks like this:

    source --> filter 1 --> filter 2 --> sink

Conceptually the data is a *stream*, you can think of it
as an endless string (or at least a string whose length
you don't know).

More practically, the data is a series of small strings
called *chunks*.

The *source* is a function which emits a chunk every time
it is called (and nil when it is done, like your usual
Lua iterator).

A *filter* is a function that takes a chunk and returns a
modified chunk. Usually it will correspond to the same data
but *not always* (see "One last important change" in the
page I pointed to).

A *sink* is the "end of the road" for chunks. It takes
chunks and does something with them. If you know functional
programming, the concept is somewhat similar to folds.
A good example of a sink is the popular one that takes
chunks in and stores them in a table.

So what is a *pump* then? Well all those blocks describe
a system through which data flows, but you have to
*actually* make data flow through the system. A pump does
just that: it takes *the system* (actually a source and a
sink since filters are attached to the endpoints earlier)
and sends some or all data from the source to the sink
through all the filters.

It turns out I like that model a lot. It is very similar
to Dataflow Programming. Read the webpage, it's worth
your time :)

-- 
Pierre Chapuis