What is an extension language (was Re: need example code)

[Bennett Todd <bet@...>]

[ warning: this is a wandering bunch of musings on programming language style
  and ways of using languages, not a whole lot of meat here ]

1999-01-21-01:36:38 Jay Glascoe:
> Also, at one point I thought I knew what "extension language" meant: I
> use Python all the time and implement compute-intensive tasks as C
> extensions (dynamically loaded, it's real cool).  So, I thought an
> extension language was something where you could extend the
> (presumably high level) interpreter with (low level) compiled code.

I think that use is what most folks are gonna be calling an extensible
language. 

Which Lua is.

When I think "extension language", I think of things like Emacs's Lisp, in
which extensions to the basic app can be written. TCL was originally designed
to be an extension language. Performance wasn't a design goal, and as it turns
out "everything's a string" isn't an attractive fundamental abstraction for
programming-in-the-large, though it works fine for tiny stuff, and was
fabulous for the two tools (or libraries) Tk and Expect. But TCL is slow and
huge, that makes most folks loathe to tack it onto any but the slowest and
hugest of apps. So for "extension language" I think something kinda like
"embedded language", in the sense that the language gets embedded in some
other app. But that usage runs afoul of "embedded code", which most folks use
to mean "code that runs in toasters and like boxes, on raw iron without an
OS", so I shouldn't use that phrase:-).

For lots of kinds of programming, Perl is my favourite language; I find I can
express myself with less strain and feeling of constraint in that language;
the many different idioms make for fluid expression --- for me. And in the
last few years it has successfully borrowed the magnificent feature that
Python in turn borrowed from the Lispers: dynamically loadable extensions
elegantly coupled with O-O namespace management.

C remains my favourite "portable assembler"; I think it's wildly the best
choice of language to e.g. implement another programming language in. Or write
small fast simple system utilities. Or write extensions for a nicer,
higher-level language that do specific performance-demanding subtasks
efficiently.

Lua is attractive to me for the case where _most_ of the work can and should
be done in C, and all you need is a lightweight programming language atop that
to provide limberness and flexibility. I'm hoping to write a procmail
replacement, for instance, by gluing Maildir delivery (already written) and
message parsing (work in progress) underneath Lua; let the equivalent of
dotprocmailrc be written in Lua, a way sexier language --- and Lua is still so
small and fast that the end result can hope to be as tiny and as
breathtakingly fast as procmail. I hope.

Another project I dream of is a super-lightweight database peruser. I just
adore the user interface of the builtin database app in the HP 100/200LX
palmtop. So much that I'm constantly running into limits, namely in the size
of database it can manipulate. I'd love to be able to prepare much bigger
databases --- multi-megabyte anyway --- on a workstation, and download 'em to
my palmtop, and have a super-fast lightweight browser with the look-n-feel of
the builtin DB. As long as I'm at it, why not use a really good efficient
compact format, say huffman coding the data with fixed trees; I've got all the
CPU in the world to prepare the data, I just need a tiny efficient browser.
Doing the entirity of such a thing in C would be a drag and a hassle; once you
get used to nice garbage-collected languages with O-O features and useable
strings and so on, it really sucks to go back. But doing the
performance-critical bits as C libraries (mostly already done by other folks,
hooray!) and the top-level app design and implementation in Lua would rock.

So I guess I see Lua as partly an extension language and partly an extensible
language. It's also a terrific candidate for an embeddable language, in the
conventional sense of a language for writting embedded apps. Small _is_
beautiful.

Until I met Lua I was leaning towards Scheme for this sort o' role, but
always ran into a couple of barriers: first, while Scheme is so elegant in
theory it's scary, actually looking at code makes my eyeballs ache. A bowl of
fingernail parings. And second, while it _seems_ like you oughta be able to
enjoy a full-featured high-performance Scheme interpreter in a couple dozen
KB of executable, somehow the actual ones end up being a couple hundred KB
--- fine for many apps --- but procmail is a 50-odd KB executable on a lot of
platforms. Quadruple that and you will significantly cut its delivery speed at
the extremes, and the extremes are where speed counts.

-Bennett