Lua Versus Unreal Script
 |
||
|
Lua Versus Unreal Script |
|
UnrealScript is a programming language designed to map naturally onto the needs of game programming within [Epic Games]' Unreal Engine.
UnrealScript is bytecode based: the code is compiled into a series of bytecodes similar to p-code or the Java bytecodes. This way UnrealScript remains platform independent and eases porting to other platforms like the Mac or consoles. UnrealScript is also a garbage collected language. All objects and actors in Unreal are garbage-collected using a tree-following garbage collector similar to that of the Java VM.
The reasoning for investigating the relative performance of UnrealScript vs. Lua is to see if Lua can be used in a game engine in the same fashion that UnrealScript has so successfully been used across Unreal Engine based games.
Lua and UnrealScript are pretty different languages. They are both compiled to bytecodes, executed through a virtual machine, offer a pointerless environment with automatic garbage collection, and provide a secure execution sandbox. This is where their similarities end.
UnrealScript is a strongly typed language like Ada and C++. This provides the benefit of catching unintentional type errors at compile time while sacrificing some flexibility. Lua on the other hand is dynamically typed. In practice this can lead to some wasted debugging time in some cases, but is not as big a problem as it first seems. Still in my opinion Lua could use a lint like pass before compilation.
UnrealScript also supports major concepts like time, state, properties, and networking through explicit language features. This removes alot of complexity that would be required if implemented at the script level. Naturally Lua lacks these features as it's meant for general purpose use, but it's support for meta-mechanisms could be used to provide similar features.
Based on the popularly quoted performance approximation, Lua is 10x slower than C code. Epic claims that UnrealScript is a 20x performance hit to C++ code. Below I've ported several benchmarks from [The Great Win32 Computer Language Shootout] to UnrealScript in an attempt to do a direct performance comparison.
To do all timing I used wrote a small testing function in Lua which times the tested code with os.clock() several times in a loop. It then throws out the fastest and slowest result and averages the times to get a final timing. UnrealScript is more complex as none of the functions documented for timing code report consistant results. So I instead launch them via UCC and a [Commandlet]. To remove the UCC launch overhead I time an empty unreal function which I then subract from the tests.
The Lua release used is 5.0.2 (the latest stable release), UnrealScript was running within Unreal Tournament 2004 build 3323, and the C++ test was done in Microsoft Visual C++ .NET 2003.
All tests performed on an Intel P4 2.8GHz with 1GB of memory.
The Lua5 benchmark for Array Access in the shootout has a bug (a superfloues "+ 1" in Line 10 ). I've corrected it for the test below.
// -*- mode: c++ -*-
// $Id: ary3.g++,v 1.2 2001/06/20 03:20:02 doug Exp $ // http://www.bagley.org/~doug/shootout/
#include <iostream>
#include <vector>
using namespace std;
int main(int argc, char *argv[]) {
int i, k, n = ((argc == 2) ? atoi(argv[1]) : 1);
typedef vector<int> ARY;
ARY x(n);
ARY y(n);
for (i=0; i<n; i++) {
x[i] = i + 1;
}
for (k=0; k<1000; k++) {
for (int i = n - 1; i >= 0; --i) {
y[i] += x[i];
}
}
cout << y[0] << " " << y.back() << endl; }
Results:
static final function string ArrayTest( optional int n ) {
local int i, k;
local array<int> x;
local array<int> y;
if ( n == 0 ) {
n = 1;
}
x.Length = n;
y.Length = n;
for ( i = 0; i < n; i++ ) {
x[i] = i + 1;
}
for ( k=0; k<1000; k++ ) {
for (i = n-1; i >= 0; i--) {
y[i] += x[i];
}
}
return ( y[0]$ " " $y[n-1] );
}
Results:
UnrealScript has protection against runaway loops which is triggered by the test at over 3000. So this is all we get to go on for this test.
function array_test( n )
local x, y = {}, {}
for i=1,n do
x[i] = i
y[i] = 0
end
for k=1,1000 do
for j=n,1,-1 do
y[j] = y[j] + x[j]
end
end
return y[1] .. " " .. y[n]
end
Results:
Lua is performing at around 1/7th the speed of C++ for n=1000 and 1/40th for n=7000. It is about 24% faster than UnrealScript.
Please comment on the results and add your feedback here.
* I did the same array test in an Intel P4 2G (with 512 Mbytes RAM) and got 0.31 sec for n=1000 and 0.94 sec for n=3000. Is my machine so much faster than yours? (-- Roberto)
* Doh... it was something stupid. My profiler was instumenting lua.exe. I re-ran the tests and put my results above. I thought those numbers seemed weird. My new problem is finding an accurate timer in Unreal. Clock/UnClock? is unreliable on my box -- Tom
* In unrealscript you can use the stopwatch() function as a timer. It's not bug-free (in build 3355) so loop it and the code about 20 times to get a larger set of results. You can disable the loop runaway check with -norunaway param. -- Switch`
* If you used Clock and UnClock? to log the results, keep in mind that the stuff that comes out is actually is milliseconds and not seconds. So if you clock(f) and then unclock(f), f is in miliseconds. I am unsure if that was taken into consideration. -- Solid Snake
* Clock uses CPU cycles and isn't safe, it wraps around. So you can only use it for really small timings. Another thing to be aware of is that there's an start-up "issue" with the UnrealEngine?, the whole environment needs to be set-up and stuff like that. At the beginning UnrealScript is rather slow, running the same code a 2nd time gives better results, or even waiting a couple seconds before running the benchmark also has better results. In the real world this start-up issue is not a problem, it just might skew the benchmark results. I don't know about Lua, but UnrealScript is not optimized during compiling. Writing ++i is faster than i++. Ofcourse these things shouldn't matter because the average users wouldn't know that either. -- elmuerte
* when you do performance-timings, you should _NOT_ take the average of all the runs you time, but only THE BEST one: it is safe to assume that the actual execution time is the same each time around, so the only differences you get are because the OS-Scheduler is interfering. Thus the time that comes closest to the "true" execution time of your script is the one with the shortest runtime.
^^^^^^ Beware this advice. In my experience this is quite misleading. For example you will then assume the most lightweight execution portions of your application run to be the best performance, this is plainly misleading. You _should_ average your performance testing over a number of identical, reproducable runs, so that you have comparative benchmarks to test against. There are far more things than just the OS-Scheduler that can effect the performance of a script run. So beware the above advice. -- Grover.