Author | Topic: Problem with round | |
---|---|---|
Adrian Wykrota | Problem with round on Fri, 26 Jun 2015 11:49:02 +0200 Version 1.90.355 PROCEDURE Main LOCAL uVal := 0 PRIVATE cFormula := "(14.086600-14.286100)*1000" ? "Formula : " + cFormula uVal := &cFormula ? "Value :" + var2char(uVal) uVal := ROUND(uVal, 0) ? "Round :" + var2char(uVal) INKEY(0) RETURN Any workaround this ? | |
Andreas Gehrs-Pahl | Re: Problem with round on Sat, 27 Jun 2015 03:52:02 -0400 Adrian, If you use str() instead of Var2Char() and specify enough decimals, you will see that the display value of -199.5000 is already rounded, as the actual result of the calculation is something closer to this: -199.4999999999987, and the correct result of Round(-199.4999999999987, 0) is just -199.0! ? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700 >Any workaround this ? As a workaround, you could do simple two-step rounding, like this: Round(Round(uVal, 6), 0) -- or more logically: uVal := Round(&cFormula, 6) ==> -199.500000 uVal := Round(uVal, 0) ==> -200.00 Of course the value of 6 (decimals) for rounding the result is just an arbitrary value suggested by me, as you use operands with six decimals, and you could as well use 5, 2, or even 1 decimal (in this particular case). But using the same precision as the operands have is always a good starting point. This issue is caused by the way decimal floating point numbers are represented and manipulated in binary format, using the IEEE 754 standard. And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the limitation of the IEEE 754 standard will be what you have to live with. Andreas Andreas Gehrs-Pahl Absolute Software, LLC phone: (989) 723-9927 email: Andreas.GP@Charter.net web: http://www.Aerospace-History.net | |
Adrian Wykrota | Re: Problem with round on Mon, 29 Jun 2015 14:04:01 +0200 I will try.Thank you. Użytkownik "Andreas Gehrs-Pahl" napisał w wiadomości grup dyskusyjnych:1qtzzb5wraagp$.cjkaszrgu42o.dlg@40tude.net... Adrian, If you use str() instead of Var2Char() and specify enough decimals, you will see that the display value of -199.5000 is already rounded, as the actual result of the calculation is something closer to this: -199.4999999999987, and the correct result of Round(-199.4999999999987, 0) is just -199.0! ? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700 >Any workaround this ? As a workaround, you could do simple two-step rounding, like this: Round(Round(uVal, 6), 0) -- or more logically: uVal := Round(&cFormula, 6) ==> -199.500000 uVal := Round(uVal, 0) ==> -200.00 Of course the value of 6 (decimals) for rounding the result is just an arbitrary value suggested by me, as you use operands with six decimals, and you could as well use 5, 2, or even 1 decimal (in this particular case). But using the same precision as the operands have is always a good starting point. This issue is caused by the way decimal floating point numbers are represented and manipulated in binary format, using the IEEE 754 standard. And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the limitation of the IEEE 754 standard will be what you have to live with. Andreas Andreas Gehrs-Pahl Absolute Software, LLC phone: (989) 723-9927 email: Andreas.GP@Charter.net web: http://www.Aerospace-History.net | |
Adrian Wykrota | Re: Problem with round on Mon, 29 Jun 2015 14:07:12 +0200 It's working PROCEDURE Main LOCAL uVal := 0 PRIVATE cFormula := "(14.086600-14.286100)*1000" ? "Formula : " + cFormula uVal := &cFormula ? "Value :" + var2char(uVal) uVal := ROUND(uVal, 6) ? "Round #1 :" + var2char(uVal) uVal := ROUND(uVal, 0) ? "Round #2 :" + var2char(uVal) INKEY(0) RETURN Użytkownik "Andreas Gehrs-Pahl" napisał w wiadomości grup dyskusyjnych:1qtzzb5wraagp$.cjkaszrgu42o.dlg@40tude.net... Adrian, If you use str() instead of Var2Char() and specify enough decimals, you will see that the display value of -199.5000 is already rounded, as the actual result of the calculation is something closer to this: -199.4999999999987, and the correct result of Round(-199.4999999999987, 0) is just -199.0! ? "Value :" + str(uVal, 20, 15) ==> -199.499999999998700 >Any workaround this ? As a workaround, you could do simple two-step rounding, like this: Round(Round(uVal, 6), 0) -- or more logically: uVal := Round(&cFormula, 6) ==> -199.500000 uVal := Round(uVal, 0) ==> -200.00 Of course the value of 6 (decimals) for rounding the result is just an arbitrary value suggested by me, as you use operands with six decimals, and you could as well use 5, 2, or even 1 decimal (in this particular case). But using the same precision as the operands have is always a good starting point. This issue is caused by the way decimal floating point numbers are represented and manipulated in binary format, using the IEEE 754 standard. And as Xbase++ doesn't have BCD (Binary-Coded Decimal) math routines, the limitation of the IEEE 754 standard will be what you have to live with. Andreas Andreas Gehrs-Pahl Absolute Software, LLC phone: (989) 723-9927 email: Andreas.GP@Charter.net web: http://www.Aerospace-History.net |