[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Tick data - data scrubbing



PureBytes Links

Trading Reference Links

You know I was out at a site that sold tick data and was reading about how
they scrub their data from head to toe.  This makes no sense to me to test
your systems on their intraday data becauase as any system trader who has
spent time running the same system on different feeds (say Quote.com or BMI,
DTN and TRAD) can tell you, differences in the data can have a HUGE impact
on the system results.  You will never get the kind of real-time clean data
from vendors that it sounds like they provide -- darn near perfect.  Thus,
unless you have the same tick cleaning algorithim they have cleaning data as
it arrives on your box, which you don't, then your results on your real time
data will vary from backtests on their sterile-clean data.  For example
let's say you program stops into your system.  Bad ticks especially in the
stock market which happen every day, can trigger those stops and erroneously
impact system results.  Most stock trading systems that rely on data from
most real time vendors (who don't clean their data before sending it out)
shouldn't use stops or reference the high and low of the bar for this
reason -- it's simply not reliable.  Small errors outside the bid and ask
can effect not only indicators and when they flip but also rules that
trigger a signal.  I've seen it happen over and over and I continue to see
this between my feeds.  What's worse is this company that sells data even
merges data from different feeds so your data isn't going to be like any
body elses and it certainly isn't going to representative of your data
coming in from your vendor.

A far better solution, IMO, is to maintain your own data from your vendor or
the raw data, apply some general tick cleaning scrubbers (like ticks that
fall 2% outside previous tick and so on) to an ASCII data file, and use that
to do your backtests.

Brian


-----Original Message-----
From: Chris Evans [mailto:evanscje@xxxxxxxxxxxxx]
Sent: Tuesday, March 12, 2002 1:45 PM
To: Omega List
Subject: Tick data


thanks to everyone who offered to help me get historical tick data...