Alaska Software Inc. - Peculiar (or bug) Int() function
Username: Password:
AuthorTopic: Peculiar (or bug) Int() function
Itai Ben-ArtziPeculiar (or bug) Int() function
on Fri, 18 Dec 2015 13:07:04 -0800
Try this:
I := 541.41  
I += 41.41   582.82
I -= 82.82   500.00
J := Int(i)   499.00

-Ben
Andreas Gehrs-Pahl
Re: Peculiar (or bug) Int() function
on Fri, 18 Dec 2015 19:09:23 -0500
Ben,

>Try this:
>I := 541.41  
>I += 41.41   582.82
>I -= 82.82   500.00
>J := Int(i)   499.00

This is a common problem with IEEE Floating Point math and can easily be 
fixed by using Round() to get the correct values.

Try this instead:

Procedure Main()
LOCAL nI := 541.41
LOCAL nJ := 0
   QOut(str(nI, 18, 13))    541.4100000000000

   nI += 41.41
   QOut(str(nI, 18, 13))    582.8200000000000

   nI -= 82.82
   QOut(str(nI,  7,  2))    500.00 <== Display value Rounded by Str()!
   QOut(str(nI, 18, 13))    499.9999999999999 <== Actual value!

   nJ := Int(nI)            Using Int() here is a bad idea!
   QOut(str(nJ, 18, 13))    499.0000000000000

   nJ := Round(nI, 0)       Using Round() works much better!
   QOut(str(nJ, 18, 13))    500.0000000000000
return

Hope that helps,

Andreas

Andreas Gehrs-Pahl
Absolute Software, LLC

phone: (989) 723-9927
email: Andreas.GP@Charter.net
web:   http://www.Aerospace-History.net
Thomas BraunRe: Peculiar (or bug) Int() function
on Mon, 21 Dec 2015 10:02:55 +0100
Itai Ben-Artzi wrote:

> Try this:
> I := 541.41  
> I += 41.41   582.82
> I -= 82.82   500.00
> J := Int(i)   499.00
> 
> -Ben

In Addition to Andreas' answer I can also recommend this pages:

http://floating-point-gui.de/

Or, if you want it more detailed and mathematical:

https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

Thomas