Re: Is "scripting" truly Lua's future?

["Mark Meijer" <meijer78@...>]

That passage from Roberto's interview stuck with me too :) That's not
to say I was shocked or anything, obviously Lua is extremely well
suited for embedded use (the intended meaning here being, embedded in
a host app). It's true that right now, there are some issues when
trying to use Lua as an extensible language (as in embedding vs.
extending), i.e. writing apps primarily in Lua with support of
extension libraries. Which is why this currently is not the
"mainstream relevance" of Lua, as Roberto well put it a couple of
posts ago.

No "standard" set of libraries, can be difficult to get libraries to
work on some platforms, no preprocessor, lacking compile-time
validation, etc. But all of these problems can be, and are being,
solved. And I don't mean stopgap measures. I mean, if this continues,
then I think Lua (or rather, some "complete" Lua distribution) can be
at least as valid a choice for application programming as any. Imho,
the idea that Lua, in principle, somehow isn't suitable for this, does
not do justice. I don't think that's what Roberto meant to say, but I
just feel his statement about the future of Lua might be true only in
part. I'm sure it will be scripting, but I don't see why it can't also
be as a primary implementation language. It might not "dominate the
market" as such, but I think it can stand on its own two feet well
enough.

I don't see it as a "wrong way" (in Matt's words) of using Lua, when
using Lua as an application programming language. For my part, this is
exactly at the heart of the recent discussions about LuaRocks and Lua
for Windows, and likely a key motivation behind most "Lua
distributions" (for today's purpose, this phrase includes LuaRocks :P
). It is precisely the appeal of using Lua, with a complete set of
libraries, as the primary language to write apps in. I hope this
*will* have mainstream relevance in the future of Lua (albeit *not* at
the cost of mainstream embedded use, unlike python), and I think it is
well within the realm of possibility. I see efforts such as LuaRocks,
LfW and others, as great steps in that direction, regardless of the
different constraints and approaches taken.

Also, is the Kepler project not a case in point? Unless I'm sorely
mistaken, the Kepler project is meant to provide a Lua-based platform
for web application development. Are web applications not
applications? I know it's often called "server side scripting", and
technically Lua is in this case embedded in the webserver or plugin.
But on the other hand, the webserver itself may very well be
implemented in Lua (or an extension library), and also, I wouldn't
make the mistake of calling a web application a script.

When I think about what application development boils down to more and
more these days, it is, in an oversimplified nutshell, the glueing
together of functionality which resides in libraries. This is nothing
new of course, but the ratio of library code vs. glue code is getting
ever more extreme. Fifteen years ago, I wrote my own cheesy 3D-library
in C and assembly, down to the function that rendered gouraud-shaded
triangles on my screen in the all-popular mode-X. Who in his right
mind does this today?

When you're writing in "pure" C++ today, think about what remains of
your application (not talking about system programming here) when you
take out all the libraries, let's say all the code not written by you
(or your team). Glue and business logic. How is it that we're still
using C++ for that? I'm no longer a professional developer (by
choice), but it's still one of my hobbies, and I am desparately trying
to switch entirely to Lua for writing my hobby apps (in case nobody
noticed yet ;). Call me weird, but it makes a lot of sense to me.

Is not the entire .NET effort based on this very notion (and I quote
one sentence from the very first paragraph of
http://en.wikipedia.org/wiki/.NET_Framework ) - "It includes a large
library of pre-coded solutions to common programming problems and a
virtual machine that manages the execution of programs written
specifically for the framework." That sums it up nicely. Ok, .NET or
say Java have *a lot* of advanced features for what is termed
enterprise applications, and are often used for that. So where do
these advanced features come from? Even if they call it "runtime"
(java runtime, common language runtime, C runtime, runtime this
runtime that)... are runtime libraries not libraries? This is not to
say we should be building a framework of such scope for Lua. But it's
certainly possible.

Many Lua modules consist of bindings to existing libraries. Libraries
that are otherwise finding use in applications written in a "regular"
application programming language (which, if the language runs on a VM,
including aforementioned .NET and Java, requires bindings of their
own). How is that use of libraries different from use of the same
libraries in Lua? Libraries for networking, database access, web
access, serialisation, multimedia, user interfaces, hardware
abstraction, containers/data structures/algorithms/templates (yes STL
too), maths, compression, encryption, authentication, text
manipulation, image manipulation, DSP. For goodness sake even complete
text editors and web browsers are being provided as library
components. And let's not forget 3D engines, AI and pathfinding,
animation, physics, input devices, 3D audio, etc. A great many of
these are already available to Lua in some form or another. In other
words, what makes Lua such a poor choice to write your glue and
business logic in?

In short, in this regard, I see no difference between Lua and any
other language typically used for application programming. Lua, the
core language, should keep doing what it's doing. But on top of that
(as a separate effort), I'm already seeing more and more convergence
towards creating distributions of Lua with a complete set of libraries
that can be easily deployed, and used as a development platform in its
own right. And I don't see why not :)

I'm not a python programmer, I've never written one line of python
code in my life. I'm not taking a stand as to whether Lua should be
used for embedding or extending (yes I've read the python article
about that quite some time ago). I just think Lua can be excellent for
both. And it took me this long and incoherent post to explain (sorry
it's late and I'm rambling)...

Cheers
Mark


2008/9/12 David Given <dg@cowlark.com>:
> Matt Campbell wrote:
>> If I'm designing a new application, should I
>> write the application core in C and then use Lua for scripting on top of
>> that, or should I write the application in Lua, using C extension
>> modules only where needed?
>
> Well, I did the latter for WordGrinder, partly as an experiment to see
> how well it worked.
>
> <plug> http://wordgrinder.sourceforge.net </plug>
>
> The short answer: it works awesomely well.
>
> What I have is a complete if simple word processor, written from
> scratch, in 5600 lines of Lua and 1400 lines of C (not including the Lua
> and LFS engines, of course). The C contains low-level screen drawing
> routines, UTF-8 read/write routines, various bits of contorted setup,
> and a very few specially optimised routines for doing stuff with words.
> Everything else is in Lua.
>
> Lua's talent at storing complex data structures intuitively allowed me
> to put together a data model for storing documents that's fast and
> reasonably efficient: a document is an array of paragraphs, a paragraph
> is an array of words, and each word is a table. Because arrays are
> tables, that allows me to attach methods and cached state to every
> object. For example, line wrapping is done by storing, with each
> paragraph, an array of lines, where each line is itself an array of the
> words in the line. This then gets regenerated on demand. Since drawing
> occurs at the cursor position and works up and down until the screen
> bounds are reached, this means that the bulk of the document won't get
> wrapped until you're actually looking at it --- which means it's fast.
>
> There's nothing there that's particularly hard. I could do it all in C++
> with the STL, for example. But it would have taken me a lot longer than
> a month to do, which is how long the first working version of
> WordGrinder took.
>
> It's also very easy to modify. For the most recent version, 0.3.1, I
> added a Table Of Contents feature: a dialogue that produces a
> hierarchical list of headers and allows you to jump to one. This took
> about an hour to write and is 97 lines of code, including comments and
> whitespace:
>
> http://wordgrinder.svn.sourceforge.net/viewvc/wordgrinder/wordgrinder/src/lua/addons/goto.lua?revision=121&view=markup
>
> It's also *efficient*. New with 0.3.1 is a Windows version. Because
> Windows machines tend not to come with package managers, I had to embed
> the Lua and LFS code into the executable. Since I was doing this, I also
> decided to byte-compile the Lua source code and embed that, as well, to
> produce a single, stand-alone executable. This is 266kB uncompressed
> EXE. That's the WordGrinder Lua code, the WordGrinder specialist C
> bindings, the Lua engine itself, the LFS bindings, and all the overhead
> needed for a Windows app, which is not insubstantial (including the icon
> --- all 45kB of it!). I think that's quite impressive.
>
> However, doing things in Lua also has its disadvantages. Like all
> languages, Lua has its quirks --- details below. However, the major
> architectural issue is that Lua, being a dynamic language which does
> most things at run-time, allows you to compile in typos. 0.3.1 is a
> bugfix release of 0.3 which I released when someone pointed out that if
> you did File->New the application crashed. Highly embarrassing. While
> fixing that I found another similar crash. I hope there aren't any more
> --- but it's very hard to know.
>
> Also, writing code in Lua, once you have it straight in your head, is
> *so* fast it requires iron discipline to keep the code sensible, well
> organised, and well commented. It's absolutely vital to maintain coding
> standards, naming conventions, file organisation, etc. It's very
> tempting to let your program fill up with quick hacks, which means it
> quickly turns into an unmaintainable spaghetti mess full of special
> cases. At least it's very quick to redraft large passages when you
> decide to change an algorithm or data structure; this lends itself well
> to evolutionary design, which is how I tend to code.
>
> ...
>
> That concludes the 'my experiences writing a moderately complex app in
> Lua' essay, so now on to some of the specific rants:
>
> - Lua doesn't have a preprocessor. My app was made up of multiple source
> files, and most of the time I wanted the same common header at the top
> of each source file (importing C functions, mostly). Many's the time I
> wished for #include.
>
> - the module structure is a pain when writing single applications ---
> because once you've done module(), the standard library is no longer
> available. You have to import everything with 'local print = print'
> statements. And if you forget one you get no warnings or error messages
> until it crashes at run time, either. Again, #include would have helped
> with this. Eventually I gave up on modules.
>
> - 1-based string offsets... do, actually, cause problems. I think I've
> talked about this before; the problem is string positions and string
> deltas. With 0-based offsets, they're the same. With 1-based offsets,
> they're not, which means you need *different code* depending on whether
> you're referring to something by position or by delta. This genuinely
> did cause me a lot of grief until I got everything straightened out.
>
> However, I should point out that I would *still* rather cope with Lua's
> quirks than, say, Symbian's C++ dialect, which is what I'm currently
> struggling with in my day job...
>
> --
> ┌─── ｄｇ＠ｃｏｗｌａｒｋ．ｃｏｍ ───── http://www.cowlark.com ─────
> │
> │ "All power corrupts, but we need electricity." --- Diana Wynne Jones,
> │ _Archer's Goon_
>
>