11. Conclusion

The current state of retail backtesting practice is characterised by a significant gap between the apparent sophistication of the tools and the actual rigour of the analysis they produce. The Python ecosystem has made it easy to generate a backtested equity curve, but the resulting curves are misleading.

The errors are not random, and their direction matters more than their frequency, because an error that’s frequent but symmetric is survivable while one that’s rare but one-directional is not. Most of the errors in this paper push toward overstating performance. Naive fill assumptions that ignore adverse selection and limit halts, understated transaction costs including roll friction and the way margin requirements move under stress, contaminated data that diverges between vendors and is silently rewritten over time, data whose definition changes by dated notice (session hours, trade dates, and the 23-hour US equity session due on 6 December 2026) while every bar stays well formed, the failure to model exchange rate effects on cross-currency positions, the implicit assumption that the broker and exchange are always there, and the inability of coarse bar data to resolve intraday order execution collectively produce results that are more optimistic than achievable reality.

Data errors are the exception, since they can push a research decision either way. A corrupted bar can trigger every stop at once and fail a strategy that deserved to pass. Corrections rewrite history in both directions, whether a vendor publishes them, as Databento did in 2026 for parts of its 2010 to 2015 CME history, or applies them silently with no notification or versioning, as Norgate does. A strategy wrongly kept shows itself once it is traded, but one wrongly rejected never comes back.

There is also a class of failure that operates on the reporting side of the backtest rather than the simulation side. Pre-tax returns can overstate the after-tax outcome by 30% or more for high-turnover strategies in a high marginal bracket. A nominal-currency equity curve says little about the home-currency balance the trader actually holds, and counterparty risk is silently assumed away by every backtest that does not apply a haircut for the historical base rate of venue failure. These gaps sit outside the simulation engine, so closing them means accepting that the backtest output is a different object from the live trader’s experience, and translating from one to the other.

The bar resolution problem deserves particular attention because it is so widely underestimated. Strategies can be designed to work within the constraints of daily data, which is why so many traders assume longer-term strategies are immune to intraday data requirements, and that assumption is seriously incomplete. Any strategy that interacts with the market through price-contingent orders, which includes virtually all strategies that employ risk management, is subject to the OHLC sequence ambiguity and the stop-loss concealment problem described in Section 4. For the order of a bar’s extremes, the severity of these problems scales directly with bar size: negligible at one-minute resolution, modest at hourly resolution, and potentially disqualifying at the daily level. The fill and gap assumptions follow no such scale, and a limit that fills at its price whenever the range covers it, or a stop that fills exactly at its trigger, is optimistic at any resolution. The commonest daily-bar heuristic (low first when the bar opens in the lower half of its range, high first otherwise) ordered the high and low correctly on 81% of days when I measured it against one-second paths of continuous ES futures, 4,986 sessions from June 2010 to June 2026. That is one instrument, and even there the figure flatters the heuristic, which is a coin flip on days that open mid-range (the choppy days when a stop and a target race each other) and whose errors run one way. So for such strategies daily data adds a directional bias that systematically flatters performance, on top of the noise. The trader who uses daily data responsibly must design their strategy to avoid dependence on intra-bar order resolution, and must acknowledge the resulting limitations in their performance claims.

Layered atop these mechanical problems are statistical methodology failures that afflict even technically correct backtests: insufficient sample sizes, uncontrolled multiple testing, the failure to account for regime dependence, and structural change inside the sample itself. On Kenneth French’s CRSP value-weighted market data from 1926 to 2026, a rule that simply followed yesterday’s direction earned a Sharpe ratio of 1.86 before Lo and MacKinlay (1988) documented the pattern, and 0.00 since. Those are paper portfolios with no costs or shorting limits, so the early figure is an upper bound. Yet the same century of data was the best guide I found to the next decade of the equity premium and of momentum, in a look-back test whose overlapping windows make it a description rather than a formal test. The mechanism behind an edge decides which of those cases applies, and more years of data cannot settle it. The survivorship bias inherent in selecting apparently successful strategies from a large universe of tested variants (whether that universe is generated deliberately through combinatorial search or implicitly through iterative refinement) further inflates reported performance. Taken together, the mechanical and statistical errors mean that a backtest must clear a very high bar before it constitutes meaningful evidence of a tradable edge.

The robustness testing methods commonly applied to address these concerns (Monte Carlo trade shuffling, synthetic data generation, and walk-forward analysis) provide less protection than is generally assumed. Monte Carlo trade shuffling violates the independence assumption for any strategy with path-dependent position sizing or portfolio-level risk filters, producing confidence intervals derived from trade sequences that could never have occurred. Synthetic data methods either destroy the statistical properties the strategy was designed to exploit or test generalisation within the same distribution rather than across genuinely novel conditions. Walk-forward analysis, while the most methodologically sound of the three and an essential minimum standard, is vulnerable to meta-overfitting and can only validate against conditions present in the historical record, some of which no longer exist. A window that spans a dated break averages two markets, so a pass can be carried by an era that has gone and a failure can be the market changing rather than overfitting; the windows should start and end at those breaks, with each era reported separately. The trader who treats a successful walk-forward pass as conclusive evidence of robustness has confused a necessary condition with a sufficient one.

Beyond the technical deficiencies, the ecosystem surrounding retail backtesting contributes to poor outcomes in ways that are less obvious but equally consequential. The self-reinforcing dominance of Python channels traders toward a particular set of tools and conventions that, while accessible, encourage methodological homogeneity and obscure the disciplines of risk management and portfolio construction that are essential to converting a trading signal into a viable system. The novice trader who emerges from this ecosystem is equipped with the ability to generate impressive-looking equity curves but is badly underprepared for the realities of live trading, including the psychological challenge of enduring drawdowns that are almost certainly deeper than those shown in the backtest.

Python (or any other language) has real value for backtesting, and it remains an excellent tool for exploratory research and the preliminary stages of strategy development. Daily bar data is usable too, for strategies designed around the constraints of coarse bars by people who understand and respect those constraints. But I believe that easy access to a programming language and its trading libraries is sending many aspiring traders down a disappointing path, because it encourages them to believe that the programming is the hard part and the trading will follow naturally.

The reality is the reverse: programming is the accessible part, and the trading competencies that determine success (risk management, statistical literacy, cost awareness, psychological discipline, and the judgment to distinguish a genuine edge from a statistical artefact) are far harder to acquire.

The ground also keeps moving under the data, and the next break already has a date: US stocks are due to start trading 23 hours a day on 6 December 2026, and any backtest that spans that date will average two markets unless its author notices.

A carefully constructed simulation, built on high-quality data with realistic execution modelling and rigorous statistical methodology, remains an essential tool for strategy development. But the distance between a credible backtest and what is commonly produced is enormous, and the ease with which the latter is generated has created a false sense of confidence that is, in aggregate, likely destroying more capital than it creates.