|
PureBytes Links
Trading Reference Links
|
You have just confirmed my worst fears. Metastock has effectively
destroyed their v8.01 System Tester so Metastock is no longer of any
use to the knowledgeable algorithm developer, while at the same time
making it so complicated that it takes months for experienced users
to learn how to use it. The novice doesn't have a chance. What a
fiasco.
David
--- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
<tlsprunger@xxxx> wrote:
> David, good thoughtful comments.
>
> I do things similar to what you are suggesting, but do it by making
a run,
> exporting the results to Excel, making another run, exporting to
Excel, etc
> etc. Then in Excel I can manipulate results and parameters to my
hearts
> content. You can use statistics, sort them, plot them on a 3-D
graph, etc
> etc.
>
> As for will Equis make big changes in the tester? I agree with
you, and the
> answer is no. It is my impression that the "broker module" that
was added
> was put in there by the CEO who wanted to make the testing "more
realistic".
> From what I can see, they failed miserably and he must be a lousy
trader.
> One of the big reasons for changing it at all was to allow
backtesting of
> multiple securities. If they had just stopped there.......... I
also have
> the impression they are targetting a much less experienced user now
than
> before. I get the feeling they would like to compete with TC2000
or some of
> the other "low end" programs (TC2K guys don't get mad at me. Its a
nice
> program but not worth much for system development). Hence the
big push on
> getting people to by the add-ons.
>
>
>
>
> ----- Original Message -----
> From: "equisdvc" <david_cardner@xxxx>
> To: <equismetastock@xxxxxxxxxxxxxxx>
> Sent: Thursday, December 04, 2003 9:56 AM
> Subject: [EquisMetaStock Group] Re: Bugs in v8.01
>
>
> > Tom,
> > I hit the wrong button on my mouse and the note I was writing
> > disappeared. You may have it, but I'll do it again in case you
don't.
> >
> > I lost my files on the points test due to a crash, so it will take
> > some time to generat a new set. I'll try to do that by Friday
> > evening. In the meantime some observations and comments.
> >
> > First, the fundamental design basis for the System Tester is
wrong.
> > Even if they eliminated every bug, it will still produce totally
> > irrelevent results for someone trying to develop an indicator with
> > one or more paramters. The fundamental concept being used is to
> > manipulate the parameters to maximize the portfolio return.
However,
> > if that same approach is applied next week, next month, or next
year,
> > it will generate significantly different parameter values each
time.
> > If a portfolio of different stocks is "optimized" the parameter
> > values will be different than those for the original portfolio.
In
> > short, one cannot use the parameter values on a different
portfolio
> > or at future times and have confidence that they are producing
> > reliable results.
> >
> > What is needed is a method that will generate a set of parameter
> > values that will be essentially unchanged over a period of time
with
> > the same portfolio and will be essentially the same for different
> > portfolios. Only then can the trading algorithm be trusted to
produce
> > valid and reproducible results day-to-day.
> >
> > In this situation I am refering to portfolios with a sufficient
> > number of entries to provide a statistically valid sample of the
> > entire population.
> >
> > In order to do this the objective function for the System Tester
must
> > be changed. It can no longer be simply to maximize portfolio
> > return. What must happen is this. During the calculations by the
> > System Tester where it tries every combination of parameter
values,
> > the parameter values that maximize the return for each individual
> > stock must be saved. All values for a given parameter are
averaged
> > and the standard deviation calculated. An optional step would
then
> > be to eliminate all values exceeding 2, 3, or 4 standard
deviations,
> > and then average the remaining values. These average parameter
> > values can then be used in the trading algorithm to compute
portfolio
> > returns day after day with confidence. They provide the best
match
> > of model to reality that can be achieved with that algorithm. In
> > short, the objective should be to optimize parameter values, not
> > portfolio return. It is only with optimized parameter values will
> > the portfolio be optimized over time.
> >
> > If the same test is performed one week, one month, or one year
later,
> > the parameter values should be little changed. If a different
> > portfolio of the same number of stocks were to be tested, the
> > parameter values would be essentially the same. Again, this is
the
> > best that can be achieved with a given algorithm.
> >
> > Second. As you may have guessed from the previous paragraphs
> > inclusion of secondary and tertiary minutia is counter productive.
> > They will be completely obliterated by the parameter optimization
> > procedure I have just outlined, while greatly complicating both
the
> > understanding and the application of the System Tester by the
user.
> >
> > Third. My recommendation would be to scape the existing v8.01
System
> > Tester entirely. As I mentioned above, even if it is totally bug
> > free, it will still generate totally meaningless and misleading
> > results for the user. I would further recommend that they take
the
> > existing v7.22 system Tester, encapsulate it in a shell that will
> > allow it to process a portfolio of stocks and generate optimim
> > parameter values as I have outlined above.
> >
> > Will Equis correct their mistake? I think it is highly unlikely.
> > Big egos made a monumental blunder and are unlikely to admit it
and
> > take corrective action. It is compounded by the fact that they do
> > not seem to understand what they are trying to do.
> >
> > David
> >
> > --- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
> > <tlsprunger@xxxx> wrote:
> > >
> > > David, please post what you find in the differences on points
only
> > test.
> > >
> > >
> > > ----- Original Message -----
> > > From: "equisdvc" <david_cardner@xxxx>
> > > To: <equismetastock@xxxxxxxxxxxxxxx>
> > > Sent: Wednesday, December 03, 2003 4:02 PM
> > > Subject: [EquisMetaStock Group] Re: Bugs in v8.01
> > >
> > >
> > > > Tom,
> > > > Thanks for taking the time to respond. I agree with
everything
> > you
> > > > had to say with one exception. I have just run a points only
> > > > comparison test and it did not match. Sent the results to
Equis.
> > I
> > > > had all of the settings you recommended. It took about three
> > months
> > > > to find them after I bought v8.01.
> > > > David
> > > >
> > > > --- In equismetastock@xxxxxxxxxxxxxxx, "Tom Sprunger"
> > > > <tlsprunger@xxxx> wrote:
> > > > > equisdvc, here is a summary of system test issues in 8.01
and
> > how
> > > > to get
> > > > > around them I have posted a few times. Hope it helps you.
> > > > >
> > > > > -------------------
> > > > > I spent a week or three figuring out how the MS v8 System
Tester
> > > > works. I
> > > > > had problems getting the results from 8.01 to match 7.2.
I
> > had
> > > > strange
> > > > > signals, missing trades, multiple trades off one signal,
etc.
> > Had
> > > > several
> > > > > emails, phone conversations, etc with Equis. Bottom line is
> > that
> > > > they went
> > > > > overboard in trying to make it realistic, to the point it is
> > > > unrealistic
> > > > > (my opinion). Especially if you are trying to develop and
test
> > > > systems.
> > > > > However, there are two workarounds you can use to get valid
> > results.
> > > > >
> > > > > So I won't forget what I learned I, wrote it up. Below
tells
> > how
> > > > the tester
> > > > > currently works and explains why the results give missing
> > trades,
> > > > multiple
> > > > > trades, doesn't use your full equity, etc. and therefore
> > provides
> > > > invalid
> > > > > results.
> > > > >
> > > > > Read below how 8.01 works, then look at the two methods to
get
> > > > valid results
> > > > > if you are trying to develop and test systems.
> > > > >
> > > > > Equis has told me they are looking into changing it, but no
> > commit,
> > > > and no
> > > > > timeframe for making the decision.
> > > > > If you agree with me, send or call them and tell them to
change
> > > > it. At the
> > > > > bottom, I list what I have recommended they do.
> > > > >
> > > > > ------------------------------------------------------------
----
> > ----
> > > > --------
> > > > >
> > > > > The Metastock v8.x system tester works very differently than
> > > > previous
> > > > > versions. On the up side it can test multiple securities
> > > > simultaneously.
> > > > > However, its operation has differences that are very
> > significant to
> > > > a system
> > > > > developer, and in reality are not anywhere near what would
> > happen
> > > > in the
> > > > > real world.
> > > > >
> > > > > It is important to understand these differences because the
v8
> > > > tester can
> > > > > give very different results from the v7 tester when using
the
> > same
> > > > system on
> > > > > the same security with the same time periods. In addition,
> > because
> > > > of the
> > > > > v8 "features" you can get different results comparing the
> > several
> > > > systems on
> > > > > the same security from the v7 tester. This also implies
> > > > Optimization
> > > > > results could be invalid.
> > > > >
> > > > > Later we will see how to get around these issues and
> > limitations.
> > > > >
> > > > > Here's the major differences.
> > > > >
> > > > > The v8 tester has several new options that are key. The key
> > > > differences
> > > > > result from a "Broker" module which places orders.
> > > > >
> > > > > a. "Position Limit" This is accessible from the "General"
> > screen
> > > > on each
> > > > > individual system test description. You will need to be
very
> > > > careful what
> > > > > you put in here as you will see later. In addition, if you
are
> > > > comparing
> > > > > different systems, you need to be sure that they all have
the
> > same
> > > > number of
> > > > > positions allowed or you may get unfair comparison results.
> > > > >
> > > > > b. Trading size. In the v7 tester the system traded 100%
of
> > your
> > > > equity on
> > > > > each trade. There was no option to do otherwise (except
> > > > for "points only"
> > > > > test). The v8 tester allows you to trade a constant number
of
> > > > shares, total
> > > > > transaction cost, or a percentage of equity available. We
will
> > > > concentrate
> > > > > on the "% of equity available, as it is most relevant for
system
> > > > development
> > > > > and comparison.
> > > > >
> > > > > c. Trade Price -- the v7 tester allowed you to select the
price
> > > > field
> > > > > (O,H,L,C) for entry and exit and the delay for each. In
other
> > > > words you
> > > > > could have different delays for entry and exit.
> > > > > The v8 tester does not allow different delays for entry and
> > exit.
> > > > It uses
> > > > > the same delay for all entries and exits. The v8 tester
also
> > allows
> > > > > different price fields for the long and short entries and
exits,
> > > > which the
> > > > > v7 did not. But there is more you should know.... read on.
> > > > >
> > > > > The biggest difference in v8 is the addition of a button
> > > > for "Realistic
> > > > > Market Prices". It is important to understand what happens
when
> > > > you check
> > > > > this button.
> > > > >
> > > > > To explain, let's assume you have "Position Limit" =1, "% of
> > equity
> > > > > available"=100 and the "Realistic Market Prices" button
checked.
> > > > >
> > > > > When you get an enter Long signal, the program calculates
the
> > > > number of
> > > > > shares that can be purchased using the open price on the
signal
> > bar
> > > > (day0).
> > > > > It tries to purchase that number of shares at the open
price on
> > the
> > > > next bar
> > > > > (day1). However, if the open price on the next bar (day1)
is
> > > > higher than
> > > > > the price from the signal bar (day0), the tester does not
place
> > the
> > > > order
> > > > > because you do not have enough money. Now let's say you
did not
> > > > have enough
> > > > > money. If your system is gives one day signals like you
would
> > get
> > > > from a
> > > > > cross(x,y) signal, then this trade will never be entered.
> > > > >
> > > > > If your system stays on a buy like you might have with a
latch
> > or
> > > > with a
> > > > > c>mov(c,20,e) structure, then the tester will still be on a
buy
> > > > signal on
> > > > > day1. It will then recalculate the number of shares that
can be
> > > > purchase at
> > > > > the open on day 1 and attempt to buy them on day2. If it
can't
> > buy
> > > > because
> > > > > the price is up at the open on day2 then it will not
enter. The
> > > > process
> > > > > will continue until either the price drops at the open and
you
> > get
> > > > filled or
> > > > > the buy signal goes away. Thus it is again possible that
you
> > never
> > > > enter on
> > > > > this trade signal. The other possibility is that you will
not
> > get
> > > > filled at
> > > > > 100% of your equity.
> > > > >
> > > > > Now for a moment assume that you had your "Position Limit"
set
> > at
> > > > 10, and
> > > > > your buy signal remains in effect. The tester will continue
> > each
> > > > bar to try
> > > > > to
> > > > > fill your orders as above, and will add positions until
either
> > your
> > > > buy
> > > > > signal goes away, or your equity is used up, or your
positions
> > are
> > > > equal to
> > > > > 10. So if you have a buy signal that lasts for 20 bars, you
may
> > get
> > > > 10
> > > > > positions filled where the last one may be filled on the
last
> > bar
> > > > before
> > > > > the buy signal goes away.
> > > > >
> > > > > Note: the reverse happens on enter short signals.
> > > > >
> > > > > When the system gets an exit signal (either long or short)
it
> > exits
> > > > on the
> > > > > open of the next bar if you have "Realistic Market Prices"
> > > > checked. Note
> > > > > that there does not seem to be any issues on the exits.
Since
> > it
> > > > is an
> > > > > exit, it just exits regardless of the prices.
> > > > >
> > > > > If your system is an always in the market system, and you
flip
> > from
> > > > a long
> > > > > to a short, it will exit longs on the next bar, and attempt
to
> > sell
> > > > short
> > > > > the number of shares it calculates based upon the open
price of
> > the
> > > > signal
> > > > > day. If the price on day1 is higher, it does not enter the
short
> > > > sale.
> > > > >
> > > > > So, it is key to realize that if you use "Realistic Market
> > Prices",
> > > > > Your tests are not at all realistic because you may not get
> > filled
> > > > on the
> > > > > day of the signal, or you may get filled at all, or you
may get
> > > > filled
> > > > > multiple times. Implications are:
> > > > > 1. Can't compare systems on same security.
> > > > > 2. buy sell arrows represent entry/exit dates and not signal
> > dates.
> > > > > 3. This problem applies to all securities including mutual
> > funds.
> > > > > 4. Very difficult to develop systems and see valid signals.
> > > > > 5. Can't compare performance on one security to another with
> > same or
> > > > > different systems
> > > > >
> > > > > Next let's review how the "Trade Price" from the "Trade
> > Execution"
> > > > screen
> > > > > works if you do not have the "Realistic Prices" option set.
> > > > >
> > > > > The system calculates the number of shares to buy based on
the
> > > > price field
> > > > > you choose and then tries to buy them on the price field you
> > choose.
> > > > >
> > > > > If you have "Buy Price" = Open and Delay =1, the system
> > calculates
> > > > the
> > > > > number of shares to buy based upon the Open price on the
signal
> > bar
> > > > and
> > > > > attempts to buy at the open of the next bar. If you
have "Buy
> > > > Price" =
> > > > > Close and Delay = 1 the system calculates the number of
shares
> > > > based upon
> > > > > the close of the signal bar and attempts to buy on the
close
> > of
> > > > the next
> > > > > bar. Same for Sell Price, Sell Short Price, etc. So if
all
> > you
> > > > do is
> > > > > uncheck the "Realistic Market Prices" button and use "Buy on
> > open
> > > > with
> > > > > delay=1", nothing really changes.
> > > > >
> > > > > This is not logical at all! If you decided to buy on the
open
> > > > tomorrow
> > > > > based on a signal today, only an idiot would calculate the
> > number
> > > > of shares
> > > > > to buy based on today's open! There has been a whole day's
> > worth
> > > > of price
> > > > > action transpired already and the open from yesterday
> > > > > is no longer relevant. You would logically use today's
close
> > > > to "Estimate"
> > > > > the number of shares that you "Might purchase" the next day.
> > > > >
> > > > > Because of all this it is difficult to develop and compare
> > systems
> > > > using the
> > > > > v8 tester that produce valid results.
> > > > >
> > > > > Now, it gets even worse. All this tells me the optimization
> > > > results are not
> > > > > valid in v8. Since
> > > > > all this buy, can't buy because price is up, etc etc stuff
> > occurs,
> > > > I assume
> > > > > it occurs on each optimization run. So your optimization
> > results
> > > > are not
> > > > > based on reality. In some trades, you don't get filled
because
> > of
> > > > this. In
> > > > > others you get filled late. A $.01 increase on the open
would
> > > > potentially
> > > > > cause you not to be filled. So it makes the optimization
results
> > > > > inconsistent and unreliable. When you optimize you want to
find
> > > > the best
> > > > > signals that optimize your performance. The potential non
or
> > late
> > > > fills
> > > > > because of a small price change on a given parameter run
totally
> > > > distort
> > > > > this.
> > > > >
> > > > > So what can you do to get around this to develop and compare
> > > > systems,
> > > > > compare multiple systems, etc. ?
> > > > >
> > > > > Note that the above issues do not seem to effect you if you
use
> > > > a "Points
> > > > > Only" test. In this mode apparently the v8 works just like
the
> > v7
> > > > tester.
> > > > > So you can get valid results that way.
> > > > >
> > > > > But if you wish to see an equity curve, or get a return in
%
> > then
> > > > you have
> > > > > the above problems.
> > > > >
> > > > > There are two ways to get around this and get valid results
and
> > > > signals.
> > > > >
> > > > > First, don't use "Realistic Market Prices".
> > > > > Make sure your "Position Limit" is set to 1.
> > > > > Set Equity Default size to 100%.
> > > > > Under Margin Requirements to
> > > > > Long Initial = 100%
> > > > > Long Maintenance = 0%
> > > > > Short Initial = 200% (amazingly, 200% means no margin,
but
> > > > that is
> > > > > another story)
> > > > > Short Maintenance = 101% (this means essentially no
> > > > maintenance. for
> > > > > some dumb reason they won't let you put in 100%, so you will
> > never
> > > > be able
> > > > > to trade your full equity on a short trade, only 99%.)
> > > > >
> > > > > Now you have two options.
> > > > > 1. Set all your enter and exit positions to "Close" with a
> > Delay =0.
> > > > > This will enter and exit all of your equity (except for the
> > stupid
> > > > 1% limit
> > > > > on short sales) at the close. All your signals will show
up on
> > the
> > > > charts
> > > > > correctly, your % gains will be correct and it will produce
> > results
> > > > very
> > > > > close to the v7 tester. Of course the drawback is that in
the
> > real
> > > > world,
> > > > > you probably won't enter and exit on the same bar as the
signal.
> > > > However, if
> > > > > you are mainly looking to compare systems or see signal
dates,
> > > > then this
> > > > > works.
> > > > >
> > > > > 2. Set all of your enter and exit conditions to use format
of
> > > > > Ref(enter,-1).
> > > > > Use "Price Field = Open with Delay =0". (or Close with
delay
> > of 0,
> > > > or
> > > > > whatever).
> > > > > What happens is that the condition that is true on day0 gets
> > > > evaluated as
> > > > > true on day1 (the day after the signal and the right day to
> > > > enter). The
> > > > > number of shares are calculated and purchased at the same
price.
> > > > The arrows
> > > > > show up on the chart on the entry/exit day.
> > > > >
> > > > > This is as realistic as the v7 tester and actually pretty
good.
> > > > > The downside is you have to rewrite all your systems to use
the
> > Ref
> > > > ()
> > > > > function in all the enter and exit conditions. Lots of
work and
> > > > pain.
> > > > >
> > > > > Using either of these methods should make the system tests
> > valid,
> > > > match 7.2
> > > > > results, and make the optimization results valid.
> > > > >
> > > > > What should be done is that Equis changes the tester. Here
is
> > what
> > > > I have
> > > > > recommended they do:
> > > > >
> > > > > If the "Realistic Market Prices" button is checked the
system
> > > > should adjust
> > > > > the number of shares to buy everything allowed by the
equity in
> > the
> > > > account
> > > > > at the open the next day.
> > > > >
> > > > > When the user selected options for Buy,Sell, etc are
checked,
> > the
> > > > system
> > > > > should adjust the number of shares to buy the total $
allowed by
> > > > the equity
> > > > > at the price field selected, on the bar selected by the
delay
> > field.
> > > > >
> > > > > Of course the other option is to just put it back to the v7
> > method.
> > > > >
> > > > > One other caveat, I have done any testing attempting to use
the
> > > > limit
> > > > > orders so I have no idea what happens when you try to do
that.
> > > > >
> > > > >
> > > > >
> > > > > ----- Original Message -----
> > > > > From: "equisdvc" <david_cardner@xxxx>
> > > > > To: <equismetastock@xxxxxxxxxxxxxxx>
> > > > > Sent: Sunday, November 23, 2003 7:53 PM
> > > > > Subject: [EquisMetaStock Group] Bugs in v8.01
> > > > >
> > > > >
> > > > > > I have been working with the Equis Technical Support
personnel
> > > > since
> > > > > > I purchased v7.0 in June of 2000. The support has been
> > > > excellent, so
> > > > > > when v8.0 was released, I bought it. I had developed my
own
> > > > trading
> > > > > > algorithm and was particularly interested in being able to
> > > > calibrate
> > > > > > it against a portfolio of equities with the System Tester.
> > > > However,
> > > > > > I found the number of bugs in v8.0 was overwhelming and I
had
> > to
> > > > > > revert back to using the v7.22 System Tester. My personal
> > > > observation
> > > > > > was that version 8.0 was not even qualified for alpha
testing
> > at
> > > > the
> > > > > > time of its release.
> > > > > >
> > > > > > I was delighted when v8.01 was released six months
later. I
> > > > thought
> > > > > > the bugs had been corrected. I was wrong. I discovered
that
> > > > v8.01,
> > > > > > although a big improvement, would barely qualify as a beta
> > test
> > > > > > version. I was both upset and embarrassed because I had
> > talked
> > > > two
> > > > > > friends into buying v8.01.
> > > > > >
> > > > > > With both v7.22 and v8.01 on the same computer it is easy
to
> > > > compare
> > > > > > System Tester results. The results between the two
versions
> > > > running
> > > > > > the identical algorithm are drastically different,
regardless
> > of
> > > > > > whether they are being run in normal or points-only mode.
This
> > > > occurs
> > > > > > when v8.01 has been carefully configured to reproduce
v7.22
> > > > results.
> > > > > > Bottom line is that v7.22 results make sense, v8.01
results do
> > > > not.
> > > > > > Technical support has not been able to explain the
> > differences,
> > > > nor
> > > > > > provide a workable solution. I still do not use v8.01. I
use
> > only
> > > > > > v7.22 because it is the only version that I trust.
> > > > > >
> > > > > > What I find unusual is that I can't find any other
reference
> > on
> > > > the
> > > > > > internet to the mismatch between v7.22 and v8.01 System
Tester
> > > > > > results. In my 43 years of using computers for problem
> > solving,
> > > > and
> > > > > > having written, debugged, and documented roughly 100,000
> > lines of
> > > > > > code, one of the primary requirements when modifying
programs
> > is
> > > > that
> > > > > > they must be able to correctly reproduce previously
validated
> > > > > > results. This test was obviously not applied to v8.0 or
> > v8.01.
> > > > Yet
> > > > > > it is the only way to validate the changes made to the
> > program.
> > > > > >
> > > > > > Is there anyone out there that encountered this situation
and
> > > > > > understands its significance?
> > > > > >
> > > > > > David
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > To unsubscribe from this group, send an email to:
> > > > > > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> > > > > >
> > > > > >
> > > > > >
> > > > > > Your use of Yahoo! Groups is subject to
> > > > http://docs.yahoo.com/info/terms/
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > > > To unsubscribe from this group, send an email to:
> > > > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> > > >
> > > >
> > > >
> > > > Your use of Yahoo! Groups is subject to
> > http://docs.yahoo.com/info/terms/
> > > >
> > > >
> >
> >
> >
> >
> > To unsubscribe from this group, send an email to:
> > equismetastock-unsubscribe@xxxxxxxxxxxxxxx
> >
> >
> >
> > Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
> >
> >
------------------------ Yahoo! Groups Sponsor ---------------------~-->
Buy Ink Cartridges or Refill Kits for your HP, Epson, Canon or Lexmark
Printer at MyInks.com. Free s/h on orders $50 or more to the US & Canada.
http://www.c1tracking.com/l.asp?cid=5511
http://us.click.yahoo.com/mOAaAA/3exGAA/qnsNAA/BefplB/TM
---------------------------------------------------------------------~->
To unsubscribe from this group, send an email to:
equismetastock-unsubscribe@xxxxxxxxxxxxxxx
Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
|