| Author | Topic: Problem with round |
---|

| Adrian Wykrota | **Problem with round**
on Fri, 26 Jun 2015 11:49:02 +0200Version 1.90.355
PROCEDURE Main
LOCAL uVal := 0
PRIVATE cFormula := "(14.086600-14.286100)*1000"
? "Formula : " + cFormula
uVal := &cFormula
? "Value :" + var2char(uVal)
uVal := ROUND(uVal, 0)
? "Round :" + var2char(uVal)
INKEY(0)
RETURN
Any workaround this ? |

| Andreas Gehrs-Pahl
| **Re: Problem with round**
on Sat, 27 Jun 2015 03:52:02 -0400Adrian,
If you use str() instead of Var2Char() and specify enough decimals, you will
see that the __display__ value of -199.5000 is already rounded, as the actual
result of the calculation is something closer to this: -199.4999999999987,
and the correct result of Round(-199.4999999999987, 0) is just -199.0!
? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700
>Any workaround this ?
As a workaround, you could do simple two-step rounding, like this:
Round(Round(uVal, 6), 0) -- or more logically:
uVal := Round(&cFormula, 6) ==> -199.500000
uVal := Round(uVal, 0) ==> -200.00
Of course the value of 6 (decimals) for rounding the result is just an
arbitrary value suggested by me, as you use operands with six decimals, and
you could as well use 5, 2, or even 1 decimal (in this particular case).
But using the same precision as the operands have is always a good starting
point.
This issue is caused by the way decimal floating point numbers are
represented and manipulated in binary format, using the IEEE 754 standard.
And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the
limitation of the IEEE 754 standard will be what you have to live with.
Andreas
Andreas Gehrs-Pahl
Absolute Software, LLC
phone: (989) 723-9927
email: Andreas.GP@Charter.net
web: http://www.Aerospace-History.net |

| Adrian Wykrota | **Re: Problem with round**
on Mon, 29 Jun 2015 14:04:01 +0200I will try.Thank you.
Użytkownik "Andreas Gehrs-Pahl" napisał w wiadomości grup
dyskusyjnych:1qtzzb5wraagp$.cjkaszrgu42o.dlg@40tude.net...
Adrian,
If you use str() instead of Var2Char() and specify enough decimals, you will
see that the __display__ value of -199.5000 is already rounded, as the actual
result of the calculation is something closer to this: -199.4999999999987,
and the correct result of Round(-199.4999999999987, 0) is just -199.0!
? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700
>Any workaround this ?
As a workaround, you could do simple two-step rounding, like this:
Round(Round(uVal, 6), 0) -- or more logically:
uVal := Round(&cFormula, 6) ==> -199.500000
uVal := Round(uVal, 0) ==> -200.00
Of course the value of 6 (decimals) for rounding the result is just an
arbitrary value suggested by me, as you use operands with six decimals, and
you could as well use 5, 2, or even 1 decimal (in this particular case).
But using the same precision as the operands have is always a good starting
point.
This issue is caused by the way decimal floating point numbers are
represented and manipulated in binary format, using the IEEE 754 standard.
And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the
limitation of the IEEE 754 standard will be what you have to live with.
Andreas
Andreas Gehrs-Pahl
Absolute Software, LLC
phone: (989) 723-9927
email: Andreas.GP@Charter.net
web: http://www.Aerospace-History.net |

| Adrian Wykrota | **Re: Problem with round**
on Mon, 29 Jun 2015 14:07:12 +0200It's working
PROCEDURE Main
LOCAL uVal := 0
PRIVATE cFormula := "(14.086600-14.286100)*1000"
? "Formula : " + cFormula
uVal := &cFormula
? "Value :" + var2char(uVal)
uVal := ROUND(uVal, 6)
? "Round #1 :" + var2char(uVal)
uVal := ROUND(uVal, 0)
? "Round #2 :" + var2char(uVal)
INKEY(0)
RETURN
Użytkownik "Andreas Gehrs-Pahl" napisał w wiadomości grup
dyskusyjnych:1qtzzb5wraagp$.cjkaszrgu42o.dlg@40tude.net...
Adrian,
If you use str() instead of Var2Char() and specify enough decimals, you will
see that the __display__ value of -199.5000 is already rounded, as the actual
result of the calculation is something closer to this: -199.4999999999987,
and the correct result of Round(-199.4999999999987, 0) is just -199.0!
? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700
>Any workaround this ?
As a workaround, you could do simple two-step rounding, like this:
Round(Round(uVal, 6), 0) -- or more logically:
uVal := Round(&cFormula, 6) ==> -199.500000
uVal := Round(uVal, 0) ==> -200.00
Of course the value of 6 (decimals) for rounding the result is just an
arbitrary value suggested by me, as you use operands with six decimals, and
you could as well use 5, 2, or even 1 decimal (in this particular case).
But using the same precision as the operands have is always a good starting
point.
This issue is caused by the way decimal floating point numbers are
represented and manipulated in binary format, using the IEEE 754 standard.
And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the
limitation of the IEEE 754 standard will be what you have to live with.
Andreas
Andreas Gehrs-Pahl
Absolute Software, LLC
phone: (989) 723-9927
email: Andreas.GP@Charter.net
web: http://www.Aerospace-History.net |