Why Your Backtest Looks Better Than Your Account

The gap that shows up on the day you start

The backtest keeps rising right up to the last day of the tested period, then stops. That boundary is not a coincidence.

David Aronson's Evidence-Based Technical Analysis takes an unusual position for a book about chart rules: most published technical rules, tested properly, do not survive. Not because charts are useless, but because the testing that produced them cannot distinguish a real pattern from an accident.

This is not dishonesty. It happens to careful people, because the process that produces it — try a rule, adjust it, try again — feels exactly like research. The adjusting is the problem.

The fastest test: check the neighbours

A real effect is broad. If only one setting works, you have found a feature of your sample, not of the market.

So when a 20-day average is the rule, test 15 and 25 as well. If they are roughly as good, the effect is probably about medium-term trend and 20 is an arbitrary point inside it. If they collapse, the 20 was reading one particular sequence of past prices.

The rules you threw away still count

This is the part Aronson is most insistent about, and the part that is easiest to skip because it feels like bookkeeping.

The instinct is to report only the survivor — the version that finally worked. But the survivor's result is only meaningful relative to how many attempts it beat. A rule that wins after three tries and one that wins after three hundred are very different claims, and the backtest output looks identical.

Testing so that something is always unseen

Hold back data, fit on the rest, score only on the held-back part — then roll the window and do it again.

This costs you something real: the scored sample is much smaller than the full history, so the result is noisier and usually worse. That is the honest version. A backtest that uses all the data and reports a beautiful number has spent its entire sample on fitting and has nothing left to test with.

1. 1Split the history before you look at it. The last third is off limits.
2. 2Build and tune on the first two-thirds until you are done changing things.
3. 3Run once on the held-back third. Once — a second run after an adjustment makes it in-sample too.
4. 4If the result is much worse, that is the finding. Do not go back and tune until it improves.

If you are not running a backtest at all

Most beginners never open backtesting software, and this still applies to them — because looking at a chart and deciding a pattern works is a backtest, just an unrecorded one with no discipline about attempts.

Eyeballing history has every flaw described above plus one more: you cannot see the cases you skipped past. The practical translation is to write the rule down before you scan for examples, then count the failures with the same energy you spent finding the successes. That is also what a trading journal is for, and why pattern reliability is a measured number rather than a felt one.