Re: Floating point inaccuracies

[<jgiors@...>]

> ------------------------------
> Date: Sun, 28 Nov 2010 22:28:07 +0100
> From: Patrick Rapin <toupie300@gmail.com>
> Subject: Re: Floating point inaccuracies
> To: Lua mailing list <lua-l@lists.lua.org>
> Message-ID:
> 	<AANLkTi=-r4mMkv0ETA4RHT34d-_kLYROhxSBS7LNAsoC@mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> This sort of problems, not directly related to Lua but to the IEEE 754
> floating point standard, comes up regularly to the list.
> Wouldn't be possible to find a more generic solution?
> The default conversion from a number to a string, used notably in
> print function, uses 16 digits of precision (based on sprintf("%.16g")
> ).
> So my idea is the following: if two very close numbers only differ
> from their 17th digit or after, the == operator between them should
> return true.
> I know that this is much more difficult to implement and slower than a
> simple binary comparison; maybe it would be too complicated to a fast
> embedded language like Lua.
> But wouldn't be better than the current situation?
> 

One problem I thought of immediately was comparison to zero, which is
usually intended to be a check for exactly zero (often to prevent
division by zero, and only zero), e.g. a [suboptimal] unit vector
function:

function unit(x,y)
    r = math.sqrt(x*x + y*y)
    assert(r~=0,"Attempted to unitize a null vector!")
    return x/r, y/r
end

But, there is an even bigger problem with approximate comparisons:  Sort
order would break [assuming approximation applies to less/greater in
addition to equals].

It is one thing to approximate "inline" comparisons (as in typical FP
noob questions, e.g. 1+.1+.1 ~= 1.2), but it would be inexcusable to
sort numerical elements out-of-order without consideration for the
entire FP precision.

John Giors
Independent Programmer
Three Eyes Software
jgiors@ThreeEyesSoftware.com
http://ThreeEyesSoftware.com