Backtesting Without Fooling Yourself
Backtests are easy to make look good and easy to be wrong about. Their real purpose is to reject bad ideas, not to promise profit.
Backtests are easy to make look good and easy to be wrong about. Their genuine purpose is not to promise profit — it is to reject ideas that do not work before you risk money on them.
A backtest that cannot kill your idea is not evidence. It is decoration.
Include every cost
Spread, commission, and swap must all be in the model. Spread is the most frequently omitted and the most damaging for short-term strategies, where it can be the difference between a profit factor of 1.3 and one of 0.9.
Slippage should be modelled too, at least as a fixed adverse amount per trade. If your strategy's edge is smaller than a realistic slippage assumption, it does not have an edge.
Look-ahead bias
Look-ahead bias occurs when a signal uses information that would not have been available at the decision point. The classic version is using the current bar's close to enter at that same bar's close — you cannot trade a price you have not seen yet.
More subtle versions include using a moving average value that includes the current bar to make a decision at the bar's open, and normalising parameters using statistics from the full sample. All of them inflate results, and none of them are detectable from the equity curve alone.
Survivorship bias
If you test on a universe of instruments that exists today, you have excluded every instrument that failed or was delisted. Your results describe a set of survivors, which is not the set you would have been choosing from at the time.
For short-term CFD and forex strategies this matters less than it does for equities, but it still bites on indices and single-stock CFDs, where the constituent list changes over the period you are testing.
Overfitting is the default outcome
Given enough parameters and enough attempts, you can fit almost any historical period beautifully. The fitted result describes the past precisely and predicts the future not at all.
The defences are few parameters, a large sample, and out-of-sample confirmation. If a small change to a parameter dramatically changes your results, you are looking at a fitted artefact rather than a robust effect.
Strategies
Explore our library of research-driven systematic trading strategies.
Key takeaways
- Model spread, commission, swap, and slippage — an edge smaller than costs is not an edge
- Look-ahead bias means no signal may use data unavailable at the decision point
- Survivorship bias excludes the instruments that failed during your test period
- If small parameter changes wreck your results, the result was fitted, not found
- Include spread, commission, and swap in every test
- Avoid look-ahead bias so no signal uses future data
- Beware survivorship bias in your instrument universe
- More parameters means more ways to fit noise
Reporting a backtest that excludes costs and then being surprised by live results.
Reading is not verification
Take the concept you just read and turn it into explicit rules, then test it. That is the only way to know whether it actually works.
Build a strategy