Design mistakes in mixed C/C++ and Lua projects (was: How can I be sure LuaJIT is working?)

[Mike Pall <mikelu-1110@...>]

Kevin T. Ryan wrote:
> > ...
> > Also have a look at the number of Lua->C and C->Lua calls via the
> > classic Lua/C API. These are expensive and neither a C compiler,
> > nor a Lua compiler can optimize across the language border. Most
> > of the time will be wasted in the API calls.
> 
> Just wondering - if someone ran into this do you think it would be
> symptomatic of:
> 
> 1. Not truly choosing a language to implement the project in (eg., too
> much mixing between the 2 languages).
> 2. Poor implementation (e.g., probably a better way to reduce the
> number of API calls).
> 3. Something else?
> 
> Maybe it's problem specific, but I guess I'm just wondering if you
> have seen this happen to yourself or someone else and what the root
> cause was (and if any generalizations could be made from such
> experiences).

<rant>

Scenario #1:

A game company starts out with a giant, over-architected C++ code
base ("Our games will NEVER need scripting"), outgrow their config
files, toy with the idea of implementing their own scripting
language, but some guy overheard something about Lua at their
previous job.

Now they throw some expensive Lua/C++ binding generators at it
(because their C++ codebase is a big pile of dung), start writing
some code in Lua that mostly calls out to C++, see it running slow
as molasses and (of course) blame Lua.

Some other guy read about LuaJIT somewhere, so they give it a try,
but obviously don't get a speedup and (of course) blame LuaJIT.
They write angry letters to me, I try to politely explain to them
that their design leaves something to be desired, they blame the
messenger.

Then they go and write a post-mortem in a game-centric blog about
how bad Lua is, that nobody should ever use scripting for anything
and that their next stellar project will use C# with lots of
enterprise-ready frameworks. And that they're hiring, because they
got nobody with a friggin clue about C#, but management dined with
someone from Redmond and they know best, of course.

Scenario #2:

A company producing consumer gear starts a new mixed C/Lua project
with dozens of developers. They chose Lua because they heard it
was fast and small and they have very constrained memory. Neither
their architects nor most of their programmers know much about
Lua, but so what -- even kids can write Lua programs, it can't be
that hard, no?

Their first discovery is that Lua provides no OO-based class
system, so they go and invent their own. It has to interoperate
with their homegrown C class system, too. Of course they implement
everything they ever heard about OO: from multiple inheritance to
SingletonFactoryFactoryProvider. You name it -- their framework
has it!

Months later, their architects have drawn UML diagrams that fill
entire hallways, piling layers upon layers and abstractions upon
abstractions. Their drones are busy implementing all of that in C
and Lua. There's no clear distinction, one of the seniors decides
based on his gut feeling and the phase of the moon. Since everyone
only gets a tiny piece of code to implement, nobody really has a
clue what they're doing and where it fits in. They urgently need
more XML in there, too. That'll solve all of their intra-app
communication needs.

Actually, nobody knows how that damn Lua thingy really works and
cargo-cult programming ensues: Programmer X really likes
semicolons, but is not keen on indentation. Everyone happily
copies their class template. Programmer Y plays it safe and
declares all variables upfront 'local i; for i=1,100 do end'.
Ooooh, that's how it works? Everyone is busy fixing their code.
Programmer Z likes to write 'local str = ""; str = somefunc()'.
You gotta make sure that variable is really a string, you know?

Vararg functions are their best friends (extensibility, got it?),
but what on earth is a local function? XML attribute lookups --
that's what pairs() is for, no? And require() is weird, let's make
sure each use somewhere else has its own global (!) handle:
'my_abc_def = require("abc.def")'. Lua strings are even weirder,
let's ban them and wrap a couple of classes around them. Make sure
the constructor does plenty of string concatenations, too.

Many months later they get something running, but everything is
really slow. Yeah, damn Lua is slow, it MUST be Lua! They start
randomly rewriting more and more code in C, but things don't
improve much. A profiler, what's that good for, anyway?

They heard LuaJIT does miracles on fibonacci benchmarks, so they
plug it in. It crashes hard due to violations of the stack
discipline of the Lua/C API. I get a bug report, tell them to turn
on assertions (for plain Lua!) before continuing with their quest.
A dozen mails later they got it running so-so. But things ain't
faster with LuaJIT! Strange, huh?

Ok, so I get to talk to their architecture-astronauts, they show
me the brilliant code their subordinates have written, I weep in
dispair. My most polite recourse: I quote my rates. Thankfully,
I never hear back.

Two weeks later a guy from a third-world outsourcing company,
thinly veiled behind a gmail account, posts some questions to the
Lua mailing list that are awfully similar to their key problems,
simultaneously displaying complete ignorance of Lua. I'd rather
bite my tongue than answer these questions. But damn it, if they
can build a Facebook clone for $200, they gotta know their stuff!

Last I heard, they are still pumping money into that project, but
that product grew a touch-screen and now supposedly does something
completely different and is coming soon, verrrry soon -- promise!

</rant>

[All described events, names and actors are made up and completely
fictitious of course. But you know: there's a piece of truth in
every story. Sorry, had to get that off my chest ...]

--Mike