[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [amibroker] Why does this simple exploration crash when run or more than 10 tickers?


  • Date: Sun, 21 Feb 2010 13:04:16 +0100
  • From: Tomasz Janeczko <groups@xxxxxxxxxxxxx>
  • Subject: Re: [amibroker] Why does this simple exploration crash when run or more than 10 tickers?

PureBytes Links

Trading Reference Links

Hello,

Hmm... you are not giving enough details therefore way too much is left 
for guessing
Your formula should not crash, unless you are using some huge number of 
bars (1+ million?) but it is written inefficent and redundant. If you 
run your formula for 20 symbols and all bars (as you seem to be doing) 
you will create 20 rows * Number_of_bars * 22 columns (20 columns for 
symbols +name + date/time)  With say one million bars (number_of_bars) 
you will end up having 440 000 000 cells. This may be reason of running 
out of memory, not to mention that Excel won't take such table.

It is *WAY* more efficient to do it right way, i.e. using the simple code:

AddColumn( RSI( 2 ), "RSI", 1.2 );
Filter = 1;

and setting Apply to "ALL SYMBOLS".

This will make AMiBroker iterate through all symbols and will be MUCH 
faster and LESS resource consuming than using your code. And you will 
end up with table having 20 rows * number_of_bars * 3 columns 
(name+date/time+rsi value),
so with same one million bars under test you will have only 60 000 000 
cells (6 times less than your code).


You mention the dates but you don't mention the INTERVAL? So data are 
from 1998 til now BUT...
at what interval? EOD ? Intraday (1-hour?, 1 - minute ???).

You need to keep in mind that if you are using 1-minute data, assuming 8 
trading hours per day gives 120000 bars per year and 12 years gives 1.44 
million bars. Each bar is 40 bytes. So each symbol intraday 1-minute 
data for 12 years back is about 60MB of data (for quotations alone). 20 
symbols put into cache would be 1.2 gigabyte for data alone. This added 
to nearly one billion cells generated by your formula (assuming that 
your data set is as large as I have been guessing here)

Anyway use
Tools->Performance Monitor
to see free memory changes.

Best regards,
Tomasz Janeczko
amibroker.com

On 2010-02-21 12:40, ramoncummins wrote:
> Hi everyone,
>
> I am trying to extract RSI(2) data for a group of stocks since 1998, using an exploration. The aim is to end up with dates down the side, and all the tickers across the top, with respective RSI(2) values in the rows. (This is to be dumped into excel later).
>
> The code below works fine (and indeed very quickly) when you limit the number of stocks to the first ten or so, but when I run the exploration over 20 stocks, it freezes and amibroker crashes.
>
> In order to replicate, just take the code below and run the exploration on SPY ONLY between 1998 and Today. It should work fine. Then uncomment the second "batch" of tickers and run it - does this crash your machine? Any ideas as to why?
>
> I have searched the posts for answers but nothing has helped so far. I presume its a memory problem of some sort.
>
> Note that I have the following settings in preferences, Data tab:
>
> In-memory cache: 20 (I have tried 11 as well, per a post from Tomasz)
> Max Megabytes: 800
>
> Any help greatly appreciated.
>
> Ramon
>
> //--------------------------------------------
>
> Filter = 1;
>
> function indicator( Ticker )
> {
>      SetForeign( Ticker );
>      myRsi = RSI( 2 );
>      RestorePriceArrays();
>      return myRSI;
>
> }
>
> AddColumn( indicator( "INTU" ), "INTU", 1.2, 1 );
> AddColumn( indicator( "VMED" ), "VMED", 1.2, 1 );
> AddColumn( indicator( "JBHT" ), "JBHT", 1.2, 1 );
> AddColumn( indicator( "ADSK" ), "ADSK", 1.2, 1 );
> AddColumn( indicator( "LRCX" ), "LRCX", 1.2, 1 );
> AddColumn( indicator( "SIAL" ), "SIAL", 1.2, 1 );
> AddColumn( indicator( "BMC" ), "BMC", 1.2, 1 );
> AddColumn( indicator( "MAT" ), "MAT", 1.2, 1 );
> AddColumn( indicator( "MYL" ), "MYL", 1.2, 1 );
> AddColumn( indicator( "QGEN" ), "QGEN", 1.2, 1 );
>
> // Uncomment below - does it crash your machine?
> /*
> AddColumn( indicator("VOD"), "VOD", 1.2, 1);
> AddColumn( indicator("PCLN"), "PCLN", 1.2, 1);
> AddColumn( indicator("CERN"), "CERN", 1.2, 1);
> AddColumn( indicator("NWSA"), "NWSA", 1.2, 1);
> AddColumn( indicator("FSLR"), "FSLR", 1.2, 1);
> AddColumn( indicator("ILMN"), "ILMN", 1.2, 1);
> AddColumn( indicator("ORLY"), "ORLY", 1.2, 1);
> AddColumn( indicator("WCRX"), "WCRX", 1.2, 1);
> AddColumn( indicator("ADP"), "ADP", 1.2, 1);
> */
>
>
>
> ------------------------------------
>
> **** IMPORTANT PLEASE READ ****
> This group is for the discussion between users only.
> This is *NOT* technical support channel.
>
> TO GET TECHNICAL SUPPORT send an e-mail directly to
> SUPPORT {at} amibroker.com
>
> TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
> http://www.amibroker.com/feedback/
> (submissions sent via other channels won't be considered)
>
> For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
> http://www.amibroker.com/devlog/
>
> Yahoo! Groups Links
>
>
>
>
>    


------------------------------------

**** IMPORTANT PLEASE READ ****
This group is for the discussion between users only.
This is *NOT* technical support channel.

TO GET TECHNICAL SUPPORT send an e-mail directly to 
SUPPORT {at} amibroker.com

TO SUBMIT SUGGESTIONS please use FEEDBACK CENTER at
http://www.amibroker.com/feedback/
(submissions sent via other channels won't be considered)

For NEW RELEASE ANNOUNCEMENTS and other news always check DEVLOG:
http://www.amibroker.com/devlog/

Yahoo! Groups Links

<*> To visit your group on the web, go to:
    http://groups.yahoo.com/group/amibroker/

<*> Your email settings:
    Individual Email | Traditional

<*> To change settings online go to:
    http://groups.yahoo.com/group/amibroker/join
    (Yahoo! ID required)

<*> To change settings via email:
    amibroker-digest@xxxxxxxxxxxxxxx 
    amibroker-fullfeatured@xxxxxxxxxxxxxxx

<*> To unsubscribe from this group, send an email to:
    amibroker-unsubscribe@xxxxxxxxxxxxxxx

<*> Your use of Yahoo! Groups is subject to:
    http://docs.yahoo.com/info/terms/