Re: Integer division

[Jay Carlson <nop@...>]

On Nov 21, 2012, at 6:55 AM, Roberto Ierusalimschy wrote:

>> Is it a goal to have this invariant?
>> 
>>  For all numbers n1, n2:
>>  For all non-debug Lua expressions L:
>>      [[ function f(x,y) return L end ]] 
>>  There do not exist integers i1==n1, i2==n2 such f(i1,i2)~=f(n1,n2)?
> 
> Not in general. First, there may be overflows inside 'f'.

I have a bad feeling about this, but it's probably overblown and misplaced. I'm thinking out loud, not speaking from authority.

Integer overflow is pernicious in C, making the Top 25 list years in a row. http://cwe.mitre.org/top25/#CWE-190 . But the worst of the damage is due to the pointer/integer interactions at the core of C, and this is not an issue for Lua. The CERT Secure Coding Standards For C section on floating point (https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=372 ) warns about floating point imprecision and all the ways FP breaks intuition too, like (a+1)-a ?= 1.

There has not been as much scrutiny of faults and defensive coding practices for all-floating-point systems like Lua--but absence of evidence is not evidence of absence, etc. So I shouldn't assume FP's problems are more benign than those of 2s-complement integers.

Integer wraparound is pretty nasty for people not expecting it. Sign-flips are particularly bad. And the wrap is usually not desired. Another language choice would be to have normal math raise an error if integer overflow occurs. There are some useful algorithms which depend on 2s-complement behavior; perhaps if they want this they can use int32.add(a,b) or "a *! b".

I really have no idea how expensive integer overflow checking is on modern processors though.

> Second, we
> probably will have a function like 'isfloat' (but it may be a debug
> function). Otherwise, I think (hope?) the answer is yes; that is what we
> want.

If overflow raises an error, any computation completing without error will be identical to an all-float version. I think.

Jay