single precision

["Brett Bibby" <research@...>]

I posted a note here asking about a garbage collection bug a few days back and I tried a few leak detectors and they reported everything seemed okay (but of course they could be wrong).  Then, when the scripts changed slightly the bug moved into a crash elsewhere.

After days of debugging a pattern emerged: if we call a function from Lua that returns a number, and then call another function that uses numbers soon after, the crash will appear.  Although bizarre, we have a lot of trial and error that shows this seems to be the case.  The only change to the Lua distribution we did was to change Lua to use single precision floats.  In doing the change, we scoured the list and here are the sum of excerpts we used to modify Lua:

***
You should probably remove all traces of 64 bit-ness on PS2. This will save memory and possibly make it run faster. In Lua 5.0 I think you just need to define LUA_NUMBER as float in your compiler environment, as you know PS2 doesn't do hardware doubles. long is 64 bit on PS2 so change all instances of this to int. Go into llimits.h and change all instances.
***
Well, I went the whole hog and upgraded to Lua 5.0 and then shifted the 64-bit numbers in the code to 32. FYI, 5.0 worked without any real modifications (the garbage collection bug I described before did not reappear straight away - although I did not conclusively test. The 64->32 bit transition wasn't as simple as it could possibly be - there are still several occurences of long and unsigned long in the code-base. Might I suggest a typedef so that is localised to a single place? Several of the occurrences mentioned '32-bit or more', suggesting that a single set of typedefs for INT32, UINT32, etc. might be appropriate (obviously with the
names munged to avoid conflicts with the Win32 typedefs with the same name).
***
The only occurrence of double was in the L_Umaxalign definition, which I changed to be: typedef union { float u; void *s; int l; } L_Umaxalign; instead of
typedef union { double u; void *s; long l; } L_Umaxalign; After that, all seems peachy. There appears to be some different behaviour
related to lua_ref, but it may just be reporting dodgy scripts that previously slipped under the radar.
***
Only reply on the PS2 dev newsgroups is currently - 'swap all 64 bit types to 32 bit', which although it solves the problem, doesn't really address the compiler issue. I will post again if there is anything more explicit to fix the compilation under GNU.
***

The most interesting thing is the comment by one poster in the second paragraph to the garbage collector and this got me wondering if this is all somehow related?  I can't imagine Lua doing anything to the floating point control word, but if I wrap our code with _controlfp(_CW_DEFAULT, 0xffff) everything works fine!

Anybody have any experience with converting Lua to float or trouble with the floating point control word?  I'm planning to get the latest Lua distro uploaded last week and try the conversion again.

Thanks,
Brett