Data conversion and merging

[James Lancaster <jimgl@xxxxxxxxxxx>]

I have a question for some data conversion expert out there .  I have been 
following this list for a couple of years and have seen very little 
addressing data build cleaning and general management.

PROBLEM:

I have several historical data base formats from 1970 to current date 1999. 
 This data covers stocks, commodities, mutual funds, and indexes .  The 
data is EOD.  I also, have tick data on all
but only several yeas back.

  The list of data vendors are prophet, momentum, metastock, dial 
data,quotes plus, tc2000, supercharts omega, omz tradestation, metastock , 
csi and windows on wall street.  Some of this data is very clean while some 
is fair and some is pretty dirty.

I am currently trying  to set up three different data bases in  formats of 
ascii , omz, and metastock  going back  to 1970.  I am using the variety of 
sources an selecting the cleanest data and converting them to these three 
formats.

The problem I am having is with the metastock  data base!

Metastock has went to a new format with only the dat. file and the old 
format uses the dat and the dop file.  I am setting up everything to the 
old format because some of my programs read only the old format an the old 
format works in all of my programs ; it just takes up a lot more drive 
space.

The problems is converting and merging the data .  I am using quotes plus 
data converter utility and the metastock  downloader.  The main problem is 
in merging the data .  The data supplied by the various vendors have 
different directory formats ; some use 250 file size directories and the 
new formats put them in categories (stocks, commodities, etc) of several 
thousand file size and some use a  alphabet format.  When using the 
Metastock downloader merging  seems to be too much for the utility even if 
you set the directories up in a directory format that should accommodate 
all formats.

My question is this.   Is there any other conversion and merging utilities 
in circulation that can work through all of these variables ,or is what I 
have, is what there is?


Jim Lancaster