Backtesting Pitfalls
1. Introduction
The barriers to entry for quantitative trading research have fallen dramatically over the past decade.
2. Data Quality: The Contaminated Foundation
A backtest is only as reliable as the data on which it is built, and the quality of freely available and even commercially distributed historical market data is far worse than most traders realise.
3. Execution Simulation Fidelity
The most fundamental question in any backtest is deceptively simple: would this trade have been filled, and at what price?
4. The Bar Resolution Problem
Every OHLC bar, regardless of its timeframe, is a lossy compression of the true price path. A bar records the open, the high, the low, and the close, but nothing about the trajectory between them.
5. Statistical Methodology Failures
Harvey, Liu, and Zhu (2016) demonstrated that the threshold for statistical significance in backtested strategies must be adjusted for the number of strategies tested.
6. The Robustness Testing Illusion
Traders who are aware of the overfitting problem described in the preceding section often turn to a family of validation techniques (Monte Carlo simulation, synthetic data generation, and…
7. The Python Ecosystem: Network Effects and Technical Constraints
Python's dominance in retail algorithmic trading is a sociological phenomenon, not the result of a deliberate technical evaluation.
8. Framework Monoculture and Methodological Homogeneity
Off-the-shelf backtesting frameworks are not neutral tools.
9. The Isolation Fallacy: Signal Generation Without Context
Novice and retail trading system developers suffer from what might be termed the signal isolation blindspot: an overwhelming focus on a single dimension of the problem, generating a profitable entry…
10. Proposed Minimum Standards for Credible Backtesting
Based on the foregoing analysis, I propose the following minimum standards for a backtest to be considered credible evidence of a potentially viable trading strategy.
11. Conclusion
The current state of retail backtesting practice is characterised by a significant gap between the apparent sophistication of the tools and the actual rigour of the analysis they produce.
12. References
1. Almgren, R., & Chriss, N. (1999). Optimal execution of portfolio transactions. Journal of Risk, 3(2), 5--39.