[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Calculation error in MS



PureBytes Links

Trading Reference Links

Allen wrote:
>An issue that also affects the database is that metastock stores data
>in the old microsoft basic 32 bit precission format. Upon loading that
>format get's converted to IEEE format single precission. We're probably
>loosing bits just in this conversion. That is then compounded by
>calculations being done in single precission.
The same number of bits are used for exponent and mantissa in both IEEE and 
the old MS format so there is no loss of precision in the conversion.
There are 23 bits for the mantissa (approximately 6 significant decimal 
digits) which is more than enough to hold raw data values of price or index 
in the data files.

It is not easy to do extended calculations which avoid issues such as the 
problem of subtraction of similar size quantities.  These things can 
produce nonsense results regardless of the precision available.   Floating 
point arithmetic also makes the use of  tests for equality problematic.

The issue is not the use of single precision but the limitations of the 
formula language provided with Metastock.   Some of you seem to be pushing 
the explorer way beyond it's capabilities.   If Metastock used double 
precision it would merely mean that it took a little longer to get into the 
same problems.

What to do when you outgrow the explorer is another debate altogether.

Lance

p.s. I've always thought the financial world to be mesmerized by numbers 
but it's not numerate.