Alaska Software Inc. - Re: Peculiar (or bug) Int() function
Username: Password:
AuthorTopic: Re: Peculiar (or bug) Int() function
Andreas Gehrs-Pahl

View the complete thread for this message in:

Re: Peculiar (or bug) Int() function
on Fri, 18 Dec 2015 19:09:23 -0500
Ben,

>Try this:
>I := 541.41  
>I += 41.41   582.82
>I -= 82.82   500.00
>J := Int(i)   499.00

This is a common problem with IEEE Floating Point math and can easily be 
fixed by using Round() to get the correct values.

Try this instead:

Procedure Main()
LOCAL nI := 541.41
LOCAL nJ := 0
   QOut(str(nI, 18, 13))    541.4100000000000

   nI += 41.41
   QOut(str(nI, 18, 13))    582.8200000000000

   nI -= 82.82
   QOut(str(nI,  7,  2))    500.00 <== Display value Rounded by Str()!
   QOut(str(nI, 18, 13))    499.9999999999999 <== Actual value!

   nJ := Int(nI)            Using Int() here is a bad idea!
   QOut(str(nJ, 18, 13))    499.0000000000000

   nJ := Round(nI, 0)       Using Round() works much better!
   QOut(str(nJ, 18, 13))    500.0000000000000
return

Hope that helps,

Andreas

Andreas Gehrs-Pahl
Absolute Software, LLC

phone: (989) 723-9927
email: Andreas.GP@Charter.net
web:   http://www.Aerospace-History.net