Re: Floating point inaccuracies

[HyperHacker <hyperhacker@...>]

On Sun, Nov 28, 2010 at 17:32, Axel Kittenberger <axkibe@gmail.com> wrote:
> been awhile dunno if I recall it correctly, but what you learn in
> coding 101 is something like this. Correct me if the comparisons are
> wrong. MYDELTA is an arbitrary tolerance value, I gave a random
> number, depending on how much crazy calculation you did before, until
> you still consider two values equal. Maybe better code adapts delta on
> the size of a/b.
>
> MYDELTA = 0.0000000001
>
> function equal(a, b)
>    return a+MYDELTA >= b and a-MYDELTA <= b
> end
>
> On Mon, Nov 29, 2010 at 12:39 AM, Tony Finch <dot@dotat.at> wrote:
>> On 28 Nov 2010, at 22:41, Andrew Lentvorski <bsder@allcaps.org> wrote:
>>>
>>> Switch to Decimal Floating Point: http://speleotrove.com/decimal/
>>
>> That just moves the problem from 1/10 to 1/3 etc.
>>
>> Tony.
>> --
>> f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
>>>
>>
>>
>
>

Yep, that's a common method. Or abs(a-b) < 0.0001, but in this case
math.abs() is a function call, so maybe not as efficient?

-- 
Sent from my toaster.