|
PureBytes Links
Trading Reference Links
|
I'm not discounting your and Fred's testimony.
Is this an acknowledgement that optimising does overfit?
BrianB2.
--- In amibroker@xxxxxxxxxxxxxxx, "quanttrader714"
<quanttrader714@xxx> wrote:
>
> I've been doing this for a long time and trust me, an OOS test is
> about an order of magnitude more important than an IS test of the
same
> size... sometimes less, but sometimes even more. And if your OOS
> period includes bearish, bullish and sideways market conditions, an
> OOS test (and inferences soundly derived from it) are as close to a
> peep into the future that you'll ever get IMO.
>
> --- In amibroker@xxxxxxxxxxxxxxx, "brian.z123" <brian.z123@> wrote:
> >
> > Fred,
> >
> > I haven't studied your I/O work, or any of your other work in
detail
> > as yet, but I would wager a bet that new traders could gain more
> > insight into I/O by *workshopping* your stuff than they can get
from
> > 90% of the books and websites etc that are out there.
> > This is pretty much true of AmiBroker across the board.
> >
> > That's a fact that needs to be stated in my book and I am just
saying
> > that for you and Tomasz so you both don't have to grapple with
modesty.
> >
> > *****************************************************************
> > OOS/WF as a peep into the future?
> >
> >
> > I'm sorry Fred but OOS/WF does not constitute a peep into the
future.
> >
> > Last night I debated with myself whether I would rebut your
arguments
> > this time around or not and my conclusion was you that would not
want
> > me to humour you.
> >
> > I will use OOS as the example, as WF is a little more complex
with
> > it's multiple, overlapping sample approach.
> >
> > OOS is just one sample, no more and no less.
> > It may be the second one taken but it is still one sample and
one
> > sample only.
> > The point is, that as a look into the future, it is the inferred
> > behaviour of the (sampling distributed)population that is
important
> > and one OOS sample a distributed population does not make.
> > I am not suggesting that we don't do it, or that I don't do it,
but
> > the simple fact is that it is a comfort blanket.
> >
> > When we look into the future we can't see a clear picture, we
can only
> > see a miasma of probabilities, and we just can not accept that.
> > It is foreign to our norms and we resist this by denial and
looking to
> > find a real time certainty that we can hang our hats on.
> >
> > I say that uncertainty is quite embraceable if her features can
be
> > vaguely discerned.
> >
> > OOS is a not a peep into the future because it is a test sample
based
> > on historical data and it provides only one equity curve based
on that
> > past history.
> > Now I don't know about you, but the positive equity curve with a
high
> > growth rate is the reason I am playing this game (lucky for me
it is
> > good fun at the same time).
> > Now the chances of that one OOS equity curve being the exact one
> > received the first time the system is traded are very slim, so
why is
> > it so important?
> >
> > Well, it's not.
> >
> > If any large data sample of trades is walked through using
random
> > selection there is an infinite number of possible equity curves,
any
> > one of which could be *the one* that the trader gets the first
time
> > they trade the system.
> > It is these equity curves that constitute the range of all
possible
> > trading futures.
> > Which one we are going to get we don't know and never will, but
we can
> > know our chances of getting any one of them,especially the bad
ones.
> >
> > I will warrant that traders remember their extreme losses more
vividly
> > than their extreme wins.
> >
> > When enough trade simulations have been conducted, all the
equity
> > curves will fall into a recogniseable distribution; one that
> > approaches a normal curve.
> > Unless a trader designs a system that never loses, the equity
curve
> > distributions will include some minus 4 or 5 standard deviation
cases
> > with a probability for each range.
> > It doesn't matter if that ruinous curve has a 1/1 billion chance
of
> > occurring, it can still occur at anytime,even on the first
trading
> > excursion with the system i.e. during the OOS test.
> > For that reason I say that in some cases we throw out perfectly
good
> > trading systems because wild chance sometimes throws up a poor
> > outcome, from a good system, during our OOS test.
> > The chances of this happening in the OOS test are exactly the
same as
> > happening in the first or any other real time trading scenario.
> >
> > If we have do one OOS test and it gets our confidence up, should
we
> > then do another one, and another one?
> > Then we could really start trading the system with confidence>
> >
> > The trading commentators I have read don't offer any firm
criteria for
> > deciding if a OOS test is acceptable or not.
> > They generally give a wishy-washy definition e.g. *If the OOOS
test is
> > within 50% of the back-test results*.
> > Why 50% and what does that mean anyway?
> >
> > My claim is that we can be definite about it.
> > If fact we can be just as definite as the margin of error
provided by
> > our sampling technique and subsequent statistical evaluation.
> >
> > In another post I will provide a simple MCS example, that to my
mind
> > demonstrates this quite clearly.
> >
> > Anyone still in doubt after that should wait for the *piece de
> > resistance*, the Random Walk, which ties it all together
irrefutably
> > for anyone who is basing their decisions on pure logic.
> >
> >
> > BrianB2
> >
> >
> >
> >
> > --- In amibroker@xxxxxxxxxxxxxxx, "Fred" <ftonetti@> wrote:
> > >
> > > OOS and/or WF Testing is not a concept invented in or for IO
so that
> > > particular piece of software is not really the issue per se
except
> > that
> > > IO has facilitated making it considerably easier to perform.
> > >
> > > "If one OOS test had a 50% drawdown it doesn't say that much
about
> > the
> > > system. It only says something about that one single OOS test
of
> > > however many samples."
> > >
> > > It doesn't ? ... It speaks volumes to me ... From my
perspective,
> > > decent in sample performance, whether or not one applies MCS
after
> > the
> > > fact, is not the end of system testing, it is only a milestone
along
> > > the way to developing a system that MIGHT be tradable ...
> > >
> >
>
Content-Description: "AVG certification"
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.14.11/542 - Release Date: 11/20/2006
|