Thursday, 07 April 2005

Floating Point Calcuations - Beware !!

Ever came across this situation ?
Try the following code and see the results ( prepare to be shocked )

Lotus Script:
Dim x as Double
x = 4.08 + 18.84 + 18.84 + 18.84 + 18.84 + 20.56
If x = 100 Then
Msgbox "X is 100"
Else
Msgbox "X is something else !!"
End if

Formula:
x:=@Sum(4.08:18.84:18.84:18.84:18.84:20.56);
@If (x = 100;
@Prompt([OK];"";"X is 100");
@Prompt([OK]);"";"X is something else !!"))
 
The Debugger & the Calculator shows that the total is 100, however the logic fails.
This can be addressed as a Floating-Point Calculation issue and its NOT a software bug ( since it has been seen in C/C++ and Excel as well ).

Some explanations from Experts in Notes Community :
 
Bruce Stapley

Only numbers with fractional components that are an exact power of 2 can be exactly represented in a binary coding system -- which is what computers use. This means very small deviations/errors occur in essentially every floating point variable in every programming language on every computer everywhere. Since Notes only stores floating point numbers, essentially every calculation you do on number fields suffers from this effect. However, the difference will assuredly be insignificant and unnoticed unless you're doing an equivalence comparison. The solution, put simply, is to round to a sufficient level of precision before comparing two floating point numbers for equivalence.

 
Stan Rogers
In LotusScript, there are precisely two levels of floating-point precision: Single and Double. Equality testing checks all bits -- any difference at any bit level is an inequality. This is true across all general-purpose computing languages, although the number of bits available to each variable type and the number of different types may vary from language to language. If you need to compare within a confidence margin, you need to round your values at your arbitrary precision depth. Some specialised languages on some platforms have similar facilities built in, but you can and should consider them oddities.
Calculators designed exclusively for decimal use often get around the problem by using binary-coded-decimal values internally, where each four-bit group represents a single decimal digit (0000 to 1001), but that is wasteful of resources and does not allow the same registers to be used for general-purpose (non-decimal) data.

 
Solution
For R5 - Round the Number to a sufficient level of precision before comparing for equivalence
For R6 - Use @FloatEq which is helpful in dealing with inexactness of floating point operations

15:30 Posted in Blog, Tips | Permalink | Comments (0) | Email this | Tags: Just talk Lotus

Post a comment