Hi,

For some reason the result of any division I do is always rounded to 2 
decimal points, e.g.

499.98 * 0.005 / 100  yields  0.02

whereas

499.98 * 0.005 * 0.01  yields  0.024999000


Can anyone please tell me if this is a setting that I can change using 
the SET command, or is it a bug in XBase++?

This is a big problem especially if you do a multiplication after a 
division as the rounding error would be magnified many folds.


Any help is greatly appreciated, thanks.

Regards,
Kelvin Wong

Hi Kelvin,

   Msgbox(str(499.98*.005/100,14,9)+;
          chr(10)+;
          str((499.98*.005)/100,14,9)+;
          chr(10)+;
          str(499.98*(.005/100),14,9))

returns:

0.024999000
0.024999000
0.024999000

I don't see anything incorrect.  The chances are that you are displaying 
the data as a string but without sufficient decimal places.

   Msgbox(str(499.98*.005/100)+;
          chr(10)+;
          str((499.98*.005)/100)+;
          chr(10)+;
          str(499.98*(.005/100)))

returns:

0.02
0.02
0.0250

If so, it's a matter of "operational precedence" coupled with display 
using str().

hth, Joe

Kelvin Wong wrote:
> Hi,
>
> For some reason the result of any division I do is always rounded to 2 
> decimal points, e.g.
>
> 499.98 * 0.005 / 100  yields  0.02
>
> whereas
>
> 499.98 * 0.005 * 0.01  yields  0.024999000
>
>
> Can anyone please tell me if this is a setting that I can change using 
> the SET command, or is it a bug in XBase++?
>
> This is a big problem especially if you do a multiplication after a 
> division as the rounding error would be magnified many folds.
>
>
> Any help is greatly appreciated, thanks.
>
> Regards,
> Kelvin Wong

IOW, the actual calculation always returns 0.024999 but str() has its 
own way of truncating such values.
Another example:

   x := 499.98*.005/100
   Msgbox(str(x)+;
          chr(10)+;
          transform(x,"9.999999999")+;
          chr(10)+;
          transform(x,"@9.9"))

returns

0.02
0.024999000
0.02

x is actually 0.024999000 - it's just the string display that's different.

-Joe

Joe Carrick wrote:
> Hi Kelvin,
>
>   Msgbox(str(499.98*.005/100,14,9)+;
>          chr(10)+;
>          str((499.98*.005)/100,14,9)+;
>          chr(10)+;
>          str(499.98*(.005/100),14,9))
>
> returns:
>
> 0.024999000
> 0.024999000
> 0.024999000
>
> I don't see anything incorrect.  The chances are that you are 
> displaying the data as a string but without sufficient decimal places.
>
>   Msgbox(str(499.98*.005/100)+;
>          chr(10)+;
>          str((499.98*.005)/100)+;
>          chr(10)+;
>          str(499.98*(.005/100)))
>
> returns:
>
> 0.02
> 0.02
> 0.0250
>
> If so, it's a matter of "operational precedence" coupled with display 
> using str().
>
> hth, Joe
>
> Kelvin Wong wrote:
>> Hi,
>>
>> For some reason the result of any division I do is always rounded to 
>> 2 decimal points, e.g.
>>
>> 499.98 * 0.005 / 100  yields  0.02
>>
>> whereas
>>
>> 499.98 * 0.005 * 0.01  yields  0.024999000
>>
>>
>> Can anyone please tell me if this is a setting that I can change 
>> using the SET command, or is it a bug in XBase++?
>>
>> This is a big problem especially if you do a multiplication after a 
>> division as the rounding error would be magnified many folds.
>>
>>
>> Any help is greatly appreciated, thanks.
>>
>> Regards,
>> Kelvin Wong

Joe thanks for the reply.

I was using the XBase++ debugger to check their values, so I assume that 
the debugger uses the Str() function to display the values.

While this cleared up my question, the focus has now been shifted to 
Str(), which, with default parameters shows values in different formats 
depending on the calculation of the values, for instance:

    x := 499.98*.005 / 100
    y := 499.98*.005 * 0.01
    msgbox(str(x)+;
      chr(10)+;
      str(y))

gives

     0.02
     0.0249990

It looks like there is still something weird going on here...

-Kelvin


Joe Carrick wrote:
> IOW, the actual calculation always returns 0.024999 but str() has its 
> own way of truncating such values.
> Another example:
> 
>   x := 499.98*.005/100
>   Msgbox(str(x)+;
>          chr(10)+;
>          transform(x,"9.999999999")+;
>          chr(10)+;
>          transform(x,"@9.9"))
> 
> returns
> 
> 0.02
> 0.024999000
> 0.02
> 
> x is actually 0.024999000 - it's just the string display that's different.
> 
> -Joe
> 
> Joe Carrick wrote:
>> Hi Kelvin,
>>
>>   Msgbox(str(499.98*.005/100,14,9)+;
>>          chr(10)+;
>>          str((499.98*.005)/100,14,9)+;
>>          chr(10)+;
>>          str(499.98*(.005/100),14,9))
>>
>> returns:
>>
>> 0.024999000
>> 0.024999000
>> 0.024999000
>>
>> I don't see anything incorrect.  The chances are that you are 
>> displaying the data as a string but without sufficient decimal places.
>>
>>   Msgbox(str(499.98*.005/100)+;
>>          chr(10)+;
>>          str((499.98*.005)/100)+;
>>          chr(10)+;
>>          str(499.98*(.005/100)))
>>
>> returns:
>>
>> 0.02
>> 0.02
>> 0.0250
>>
>> If so, it's a matter of "operational precedence" coupled with display 
>> using str().
>>
>> hth, Joe
>>
>> Kelvin Wong wrote:
>>> Hi,
>>>
>>> For some reason the result of any division I do is always rounded to 
>>> 2 decimal points, e.g.
>>>
>>> 499.98 * 0.005 / 100  yields  0.02
>>>
>>> whereas
>>>
>>> 499.98 * 0.005 * 0.01  yields  0.024999000
>>>
>>>
>>> Can anyone please tell me if this is a setting that I can change 
>>> using the SET command, or is it a bug in XBase++?
>>>
>>> This is a big problem especially if you do a multiplication after a 
>>> division as the rounding error would be magnified many folds.
>>>
>>>
>>> Any help is greatly appreciated, thanks.
>>>
>>> Regards,
>>> Kelvin Wong

If you do not specify a decimal parameter then the display will default
to the precision defined in Set Decimal.

The difference in results can be explained by the imprecision of
binary-decimal conversion with a finite number of bytes. Floating point
requires that an infinite number of decimal place be available for exact
arithmetic to take place.

Bottom line is that you need to use round() to define the precision of
any calculation to the desired precision.

Jim



Kelvin Wong wrote:
> Joe thanks for the reply.
> 
> I was using the XBase++ debugger to check their values, so I assume that
> the debugger uses the Str() function to display the values.
> 
> While this cleared up my question, the focus has now been shifted to
> Str(), which, with default parameters shows values in different formats
> depending on the calculation of the values, for instance:
> 
>    x := 499.98*.005 / 100
>    y := 499.98*.005 * 0.01
>    msgbox(str(x)+;
>      chr(10)+;
>      str(y))
> 
> gives
> 
>     0.02
>     0.0249990
> 
> It looks like there is still something weird going on here...
> 
> -Kelvin
> 
> 
> Joe Carrick wrote:
>> IOW, the actual calculation always returns 0.024999 but str() has its
>> own way of truncating such values.
>> Another example:
>>
>>   x := 499.98*.005/100
>>   Msgbox(str(x)+;
>>          chr(10)+;
>>          transform(x,"9.999999999")+;
>>          chr(10)+;
>>          transform(x,"@9.9"))
>>
>> returns
>>
>> 0.02
>> 0.024999000
>> 0.02
>>
>> x is actually 0.024999000 - it's just the string display that's
>> different.
>>
>> -Joe
>>
>> Joe Carrick wrote:
>>> Hi Kelvin,
>>>
>>>   Msgbox(str(499.98*.005/100,14,9)+;
>>>          chr(10)+;
>>>          str((499.98*.005)/100,14,9)+;
>>>          chr(10)+;
>>>          str(499.98*(.005/100),14,9))
>>>
>>> returns:
>>>
>>> 0.024999000
>>> 0.024999000
>>> 0.024999000
>>>
>>> I don't see anything incorrect.  The chances are that you are
>>> displaying the data as a string but without sufficient decimal places.
>>>
>>>   Msgbox(str(499.98*.005/100)+;
>>>          chr(10)+;
>>>          str((499.98*.005)/100)+;
>>>          chr(10)+;
>>>          str(499.98*(.005/100)))
>>>
>>> returns:
>>>
>>> 0.02
>>> 0.02
>>> 0.0250
>>>
>>> If so, it's a matter of "operational precedence" coupled with display
>>> using str().
>>>
>>> hth, Joe
>>>
>>> Kelvin Wong wrote:
>>>> Hi,
>>>>
>>>> For some reason the result of any division I do is always rounded to
>>>> 2 decimal points, e.g.
>>>>
>>>> 499.98 * 0.005 / 100  yields  0.02
>>>>
>>>> whereas
>>>>
>>>> 499.98 * 0.005 * 0.01  yields  0.024999000
>>>>
>>>>
>>>> Can anyone please tell me if this is a setting that I can change
>>>> using the SET command, or is it a bug in XBase++?
>>>>
>>>> This is a big problem especially if you do a multiplication after a
>>>> division as the rounding error would be magnified many folds.
>>>>
>>>>
>>>> Any help is greatly appreciated, thanks.
>>>>
>>>> Regards,
>>>> Kelvin Wong

Hi James,

Actually this is a situation where the extra decimal places is showing 
the correct answer.  The difference is due to precedence in the two 
calculations.

-Joe

James Loughner wrote:
> If you do not specify a decimal parameter then the display will default
> to the precision defined in Set Decimal.
>
> The difference in results can be explained by the imprecision of
> binary-decimal conversion with a finite number of bytes. Floating point
> requires that an infinite number of decimal place be available for exact
> arithmetic to take place.
>
> Bottom line is that you need to use round() to define the precision of
> any calculation to the desired precision.
>
> Jim
>
>
>
> Kelvin Wong wrote:
>   
>> Joe thanks for the reply.
>>
>> I was using the XBase++ debugger to check their values, so I assume that
>> the debugger uses the Str() function to display the values.
>>
>> While this cleared up my question, the focus has now been shifted to
>> Str(), which, with default parameters shows values in different formats
>> depending on the calculation of the values, for instance:
>>
>>    x := 499.98*.005 / 100
>>    y := 499.98*.005 * 0.01
>>    msgbox(str(x)+;
>>      chr(10)+;
>>      str(y))
>>
>> gives
>>
>>     0.02
>>     0.0249990
>>
>> It looks like there is still something weird going on here...
>>
>> -Kelvin
>>
>>
>> Joe Carrick wrote:
>>     
>>> IOW, the actual calculation always returns 0.024999 but str() has its
>>> own way of truncating such values.
>>> Another example:
>>>
>>>   x := 499.98*.005/100
>>>   Msgbox(str(x)+;
>>>          chr(10)+;
>>>          transform(x,"9.999999999")+;
>>>          chr(10)+;
>>>          transform(x,"@9.9"))
>>>
>>> returns
>>>
>>> 0.02
>>> 0.024999000
>>> 0.02
>>>
>>> x is actually 0.024999000 - it's just the string display that's
>>> different.
>>>
>>> -Joe
>>>
>>> Joe Carrick wrote:
>>>       
>>>> Hi Kelvin,
>>>>
>>>>   Msgbox(str(499.98*.005/100,14,9)+;
>>>>          chr(10)+;
>>>>          str((499.98*.005)/100,14,9)+;
>>>>          chr(10)+;
>>>>          str(499.98*(.005/100),14,9))
>>>>
>>>> returns:
>>>>
>>>> 0.024999000
>>>> 0.024999000
>>>> 0.024999000
>>>>
>>>> I don't see anything incorrect.  The chances are that you are
>>>> displaying the data as a string but without sufficient decimal places.
>>>>
>>>>   Msgbox(str(499.98*.005/100)+;
>>>>          chr(10)+;
>>>>          str((499.98*.005)/100)+;
>>>>          chr(10)+;
>>>>          str(499.98*(.005/100)))
>>>>
>>>> returns:
>>>>
>>>> 0.02
>>>> 0.02
>>>> 0.0250
>>>>
>>>> If so, it's a matter of "operational precedence" coupled with display
>>>> using str().
>>>>
>>>> hth, Joe
>>>>
>>>> Kelvin Wong wrote:
>>>>         
>>>>> Hi,
>>>>>
>>>>> For some reason the result of any division I do is always rounded to
>>>>> 2 decimal points, e.g.
>>>>>
>>>>> 499.98 * 0.005 / 100  yields  0.02
>>>>>
>>>>> whereas
>>>>>
>>>>> 499.98 * 0.005 * 0.01  yields  0.024999000
>>>>>
>>>>>
>>>>> Can anyone please tell me if this is a setting that I can change
>>>>> using the SET command, or is it a bug in XBase++?
>>>>>
>>>>> This is a big problem especially if you do a multiplication after a
>>>>> division as the rounding error would be magnified many folds.
>>>>>
>>>>>
>>>>> Any help is greatly appreciated, thanks.
>>>>>
>>>>> Regards,
>>>>> Kelvin Wong
>>>>>