Reasons for basing Ravi on Lua (long post)

[Dibyendu Majumdar <mobile@...>]

Hi,

Now that I feel I am committed to delivering Ravi as a JIT compiled
version of Lua 5.3, with optional static typing where performance
matters, I feel that maybe I ought to share why I chose Lua as the
basis for Ravi, and why Ravi came about.

I was looking for a scripting language that could be embedded in an
application. The language would be used by users to script things when
necessary. Some of these scripts could be executed in large
computations - e.g. matrix solvers - so it is important that the
scripts performed well. The language has to be very simple - although
a user would be expected to be able to program - they would not be
expected to be professional programmers.

I looked at variety of languages / implementations for this. The top
candidates were:

* Python - as it seems to be popular in the realm of numeric computing
* Javascript - The V8 implementation (and similarly Mozilla's
Javascript engine) are now heavily optimized and appear to do
extremely well in performance tests
* LuaJIT - as the performance optimized version of Lua this was a
natural choice (with Lua as the backup)
* Julia - this is the new language from MIT folks that appears to be
gaining a lot of traction
* Terra - the extension to Lua
* R - obviously R is heavily used in the industry and has large user
base in certain sectors
(And some others such as AngelScript, Pure, etc.)

I did not know much about Lua when I started my investigation. I had
heard about it but had dismissed it in the past as most of my work
life is around Java / C / C++ technologies, and I did not see the need
for another scripting language where alternatives such as Python were
available. What brought Lua to my attention was its design as a
language to be used embedded within another application - all other
languages I knew had a different goal in mind. All were designed to be
the primary language for application development.

The biggest hurdle regarding Lua was the question in my mind about its
popularity and user base - obviously Python and Javascript are more
popular languages. The choice of the language could have a massive
impact on the perceived user friendliness purely because of lack of
familiarity. This is still a question in my mind.

Python was very attractive from a user base point of view. It is well
designed, well implemented and documented. However the issue with
Python is that it is harder to use it in embedded context - the API is
more complex. I personally dislike the reliance on indentation in
Python's syntax (although my personal taste should not matter).
Performance tests indicate that Python is comparatively slow at the
type of scripts I envisaged users would write (I did not personally
benchmark this so I could be wrong here).

Javascript would have been a good choice except that I found the
implementations poorly documented, extremely large and complex.

Julia is a promising new language but the issues I found were that
it's core implementation is tightly integrated with a host of
libraries and capabilities - it is not possible to extricate the pure
language from it. Its parser is written in a custom implementation of
Lisp, there is liberal user of global / static variables in the code.
Support for embedding Julia is poor, and there are assumptions made
about the context in which Julia is being used which makes it
unsuitable (in my opinion) for embedded usage.

I found Terra to be very interesting but it is too low level for the
user base I have in mind.

R has various issues, including licensing model, performance, ease of
embedding etc.

So then the main choice seemed to be between Lua and Python. I chose
former because it was smaller and easier for me to use. Also the
availability of LuaJIT meant that performance wise there was not much
to choose from.

To be honest I had no intention of creating Ravi - I was hoping to
just use LuaJIT with Lua as a backup.

Both Lua and LuaJIT are fantastic products. The implementations are
very impressive - the design of the whole package as a library, the
use of lua_State and avoidance of globals, the rigorous and compact
coding - simply some of the best C coding examples I have ever seen.

Working with LuaJIT for while made me realize however that it is too
sophisticated for me to be able to maintain or support it. Even to
understand how it works would take months if not years of effort. Just
using it without being able to support it would be quite dangerous in
the long term, because if for some reason the development of LuaJIT
ceased then I would end up with very expensive re-write of the code
base.

I decided to solve this by maintaining compatibility with both Lua and
LuaJIT, so that I would always be able to fall back on Lua. This of
course meant that I could not use FFI as it would immediately break
compatibility. In practice for the usage I had in mind the performance
difference between Lua and LuaJIT although visible at times was not
like a factor of 10 - more like factor of 2 or 3. This is because
LuaJIT performs best when all the code is written in Lua. This is not
the case in an embedded context. Add to this the restriction that FFI
cannot be used then some of the advantages disappear.

In an ideal world I would just like to understand LuaJIT, be able to
extend / support it. But this I think is not realistically possible.
So that is why Ravi was conceived. The static typing was conceived as
a "cheap" way of getting better performance - without all the clever
stuff that LuaJIT does. I wasn't sure that it was worthwhile investing
time and effort on Ravi so I set myself the goal of quickly
establishing whether there was any real benefit in a) adding optional
static typing and b) implementing a JIT compiler. My conclusion now is
that both of them taken together achieve better performance than it
would be possible with either one alone.

The decision to implement Ravi has been very rewarding. It has allowed
me to learn much more about how Lua works. It is also my first foray
into building a JIT compiler. Thankfully LLVM has made it possible to
do this much more easily than it would otherwise have been.

Thank you for your time - both in answering my questions and allowing
me to share my experiences here.

Thanks and Regards
Dibyendu