[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Calculation error in MS



PureBytes Links

Trading Reference Links

An issue that also affects the database is that metastock stores data
in the old microsoft basic 32 bit precission format. Upon loading that
format get's converted to IEEE format single precission. We're probably
loosing bits just in this conversion. That is then compounded by
calculations being done in single precission.

Allan


--- "Genevieve, Louisa, Herve" <genher@xxxxxxxxxxxxx> wrote:
> When I replicated the experiment I created a security with, as closes
> 1.47
> and 1.5 as Orea's. I, then, read David's reply, and thought if I
> enter 1.47
> why would MS store it (internally) as 1.47000...1?David,  do you mean
> that
> at a basic binary level , MS can not store a number with infinite
> 0's, as
> 1.4700000000......and therefore has to store it as 1.47000.....1? But
> if it
> was the case , then, could  the result of a calculation EVER be equal
> to a
> fix number? In the case of Orea's problem the result of the
> calculation IS 2
> ( and not 2.000001), if it get strored as 2.000001 then this could
> never be
> equal to 2?? right??? In wich case, any formula of the type if( the
> resultat
> of a calculation=a fix number(2 for instance), Do someting, Do
> somethin
> else) will NEVER work! I am very confused.......Anyone has more
> thoughts on
> the subject?
> 
>

=====
---
Allan Havemose, Ph.D.
havemose@xxxxxxxxxx
havemose@xxxxxxxxx

__________________________________________________
Do You Yahoo!?
Listen to your Yahoo! Mail messages from any phone.
http://phone.yahoo.com