[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[amibroker] Re: Using large intra day historical data bases


  • Date: Tue, 23 Feb 2010 18:05:29 -0000
  • From: "chuck_win" <chaoy@xxxxxxxxxxx>
  • Subject: [amibroker] Re: Using large intra day historical data bases

PureBytes Links

Trading Reference Links

Keith,

I placed an order to buy those CDs. Do you have any problem to read data from them into amribroker? Is the quality of data good?

Thanks.

Charles

--- In amibroker@xxxxxxxxxxxxxxx, Keith McCombs <kmccombs@xxx> wrote:
>
> Mike, gariki, and James --
> I have stock data from PItrading.com.  1100 stocks, most going back 7 years.
> AB uses 40bytes to store one data point, OHLCV.  That means for 1min of 
> data 15,600 bytes per day, 3,931,200 bytes per year.  For 1100 stocks 
> that is 4,324,320,000 bytes, 4GB for one year.  I'm running XP64 with 
> 8GB RAM, which is about my limit without a lot more $ for RAM.  Most of 
> the testing I do requires portfolio testing with large number of stocks 
> with few trades per stock.
> 
> PITrading has a 'market' disk which contains over 235 actively traded 
> symbols, including ten futures with from 5 to 12 years of data each.  
> Check with them as to how they handle rollover date.
> 
> With my particular strategies, I see practically no difference between 
> having running with SetBarsRequired(sbrAll, sbrAll); present or 
> commented out.  I ran this simple test with the same strategy on two 
> equities: 
>   AAXJ, has data going back only to 8/18/08.
>   BHI, has data going back to early 1992.
> 
> I optimized for a variable which was set to a random number on each 
> iteration.   I also included in the strategy Buy = C < .01 AND (rest of 
> strategy);  // so no trades are generated
> EOD data was used and the Range was 8/1/2008 to 12/31/2009.
> Optimize:  1000Steps   2000Steps
>   AAXJ       3secs      7secs
>   BHI        8secs     16secs
> 
> The tests were performed multiple times giving each stock a chance to be 
> cached in RAM.
> 
> Thank you for your feedback.  But I would still like to try breaking the 
> data up into smaller time periods and would appreciate suggestions on 
> how to go about it.
> -- Keith
> 
> 
> gariki wrote:
> >  
> >
> > I have done exactly this recently; but with only about 5years of 
> > intraday 1min data. In the end once the 5year database got loaded (the 
> > initial database creation took a while) i started using that database 
> > instead of the five 1year databases and used walkforward settings to 
> > set the backtest periods to 1year intervals. Backtesting did not seem 
> > to be slow for me on the bigger database so it worked out well.
> >
> > This saved me a bit of time since i dont need to switch between 
> > databases etc.
> >
> > BTW, you mind revealing where you bought your data from and how much 
> > it costs; i am myself contemplating buying some futures 1min data from 
> > http://disktrading.is99.com/disktrading/#data1. 
> > <http://disktrading.is99.com/disktrading/#data1.> The cost seem pretty 
> > reasonable and i will have 7 more years of intraday data than i 
> > currently have.
> >
> > thanks
> > -gariki
> >
> > --- In amibroker@xxxxxxxxxxxxxxx <mailto:amibroker%40yahoogroups.com>, 
> > "Mike" <sfclimbers@> wrote:
> > >
> > > Kieth,
> > >
> > > I will admit that I have not used it. But, wouldn't a single 
> > database using QuickAFL solve the problem for you?
> > >
> > > Mike
> > >
> > > --- In amibroker@xxxxxxxxxxxxxxx 
> > <mailto:amibroker%40yahoogroups.com>, Keith McCombs <kmccombs@> wrote:
> > > >
> > > > I recently purchased 1min historical data, in ASCII format, for 
> > system
> > > > development and backtesting. Some of it goes back many years and
> > > > therefor the files can be enormous. I have NO problem with how much
> > > > disk space they take up. However, I do mind how much time is used in
> > > > back testing.
> > > >
> > > > One of my thoughts is to somehow break the data up into smaller time
> > > > periods, for example years with some overlap of November and 
> > December.
> > > > So I might have one data base that goes from 11/1/1998 to 
> > 12/31/1999 and
> > > > the next from 11/1/1999 to 12/31/2000, etc. This would 
> > allow/require me
> > > > to test one year at a time, which might also help me keep my "back
> > > > testing" separate from "forward testing" I might also consider even
> > > > smaller time periods such as quarters or months or weeks.
> > > >
> > > > Has anyone else thought about, or better yet, done this?
> > > >
> > > > Is there any efficient (time wise) way to implement this. I don't 
> > care
> > > > about disk space; it's cheap.
> > > >
> > > > Your thoughts and suggestions are appreciated.
> > > > -- Keith
> > > >
> > >
> >
> >
>




------------------------------------

**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.

TO GET TECHNICAL SUPPORT send an e-mail directly to 
SUPPORT {at} amibroker.com

TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    amibroker-digest@xxxxxxxxxxxxxxx 
    amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/