10. Proposed Minimum Standards for Credible Backtesting

Based on the foregoing analysis, I propose the following minimum standards for a backtest to be considered credible evidence of a potentially viable trading strategy. These standards are not sufficient to guarantee forward profitability, but their absence should be treated as a strong signal that reported results are unreliable.

Each standard is expanded below.

• Data quality: The data source must be documented and its known limitations disclosed. Adjustments for erroneous prints, phantom bars, missing sessions, and corporate actions must be described. Commercially sourced data must not be assumed to be clean; independent spot-checks against exchange records or alternative sources should be performed, particularly for older historical periods. For strategies sensitive to intraday extremes, the same strategy should be re-run against at least one independent vendor’s data to detect pipeline-specific artefacts. Datasets should be archived at the time the backtest is run, because vendors silently rewrite their historical records and re-running months later can produce different results from the same code. For equity strategies, the treatment of dividend and split adjustments (price-only versus total-return) must be stated explicitly, and absolute-level logic must use unadjusted prices. For futures data, the continuous contract construction methodology must be specified, and the historical trading venue regime (pit, mixed, electronic) must be identified by date, including the July 2015 closure of most of CME’s futures pits, and its implications for strategy applicability discussed.

  The definition of the data must be declared as well as its source. That means the daily-bar definition (core, full, or calendar-day session; trade date or UTC date; settlement, last print, or auction close) and a dated session registry, in which each row carries an effective-from date, a timezone of its own, the exchange notice that established it, and a separate settlement window, so that history is never restated under current hours. The trading calendar and the annualisation factor must be declared too, because until 1952 the NYSE also traded on Saturdays and a year had about 300 sessions rather than 252. Positioning and macro data, such as the CFTC’s Commitments of Traders and FRED series, must be joined on the date each figure was published rather than the date it describes, with vintages wherever revisions matter. Each instrument needs a persistent identity with dated ticker mappings, because tickers change and get reused. Archiving is only the first step: after every data update, the saved run should be replayed and the new download compared with the old one record by record, and a research log of kept and rejected strategies, with the data version each used, should be rechecked whenever a vendor publishes a correction.

• Execution model: The backtest must use a fill model that accounts for bid-ask spread (using empirical spread data where available in place of a fixed assumption), slippage as a function of order size and market volatility, realistic order sequencing, and the adverse selection inherent in limit order fills. Strategies using limit orders must model queue position or, at minimum, require price to trade through the limit level by a specified buffer before assuming a fill. The model must also handle limit-locked sessions, circuit breakers, and single-stock trading halts, rather than assuming the trader could have executed at the printed price during any such period. Limit fills inside a bar must not assume that every price in the range traded, and stops must fill at the next traded price through the level, which in a fast market is worse than the trigger. From 6 December 2026, equity strategies that trade outside regular hours must model the overnight LULD band regimes and Nasdaq’s 4:00am cancellation of overnight orders, and options strategies must apply each session’s order-type rules.

• Transaction costs: All material costs must be included: commissions, exchange fees, spread costs, roll costs for futures (including calendar spread crossing costs and roll-window market impact), financing costs for leveraged positions, and an estimate of market impact for strategies intended to operate at scale. Gross and net returns should be reported side by side at every stage of the development pipeline; the gap between them is a direct measure of how much of the apparent edge depends on optimistic cost assumptions. Costs must be dated: US brokerage commissions were fixed until 1 May 1975, and online commissions went to zero in October 2019, so a single cost assumption across a long equity sample is wrong at one end or the other. Costs, stops, and targets should be written in units that move with the market, such as a fraction of price or of recent range, rather than in ticks or points, because a rule written in ticks is a different rule in 2010 and in 2026.

• Currency effects: For any strategy trading instruments denominated in a currency other than the account’s base currency, the backtest must model exchange rate effects on trade-level profit and loss, on the value of margin collateral held in foreign currency, and on interest accrual differentials between the domestic and foreign currency. The FX overlay must be applied across the sample, at every trade, every margin revaluation, and every interest accrual, since a single conversion at the end loses the path. Performance metrics must be reported in the trader’s actual base currency.

• Margin dynamics: For any leveraged strategy, the backtest should model the time-varying behaviour of margin requirements rather than assume a fixed schedule. The combined effect of percentage-margin changes and notional drift must be acknowledged. Where the actual historical margin schedule is not available, sensitivity to percentage-margin spikes (doubling or tripling during stress periods) should be tested and reported.

• Counterparty risk: For long-horizon backtests, the implicit assumption that the broker and exchange remained solvent and accessible should be made explicit. For crypto in particular, a haircut reflecting the historical base rate of venue failure should be applied, or the strategy universe should be restricted to venues with genuine segregation and audited reserves. Stablecoin peg exposure should be flagged as a credit and reserve assumption.

• Tax reporting: Backtests must report pre-tax and after-tax returns side by side, with the holding-period regime declared and the trader’s actual marginal rates plugged in. Wash-sale equivalents and the differential treatment of futures versus securities must be accounted for where relevant. Strategies of very different turnover cannot be compared on pre-tax returns alone.

• Intraday resolution for path-dependent orders: Any strategy that uses stop-losses, profit targets, trailing stops, or other intraday-sensitive order types must be tested with intraday data of sufficient resolution to determine order fill sequence. Daily OHLC data is insufficient for this purpose regardless of the strategy’s rebalancing frequency. I’ve kept that wording absolute on purpose, as a minimum standard, and it covers path-dependent orders: any strategy whose stops and targets can share a bar. The narrow exception is a strategy on a daily or longer horizon with market-on-open entries and stops set outside the bar’s range, where the order of the bar’s extremes never decides a fill. The engine’s intra-bar convention must be stated, and optimistic and pessimistic runs reported side by side, so that the reader can see the direction of the errors as well as their rate. One-second data is enough to settle the order of the extremes: on ES one-second data from June 2010 to June 2026 (Databento), it resolved the order of the high and low on every one of 4,986 days, because the two were never closer together than about 22 seconds. That’s one instrument, and the result covers only the order of the extremes; even one-second data understates the fill and gap problems.

• Statistical rigour: Results must be reported with confidence intervals. The number of strategy variants tested must be disclosed, and significance thresholds must be adjusted for multiple comparisons using established methods such as White’s Reality Check, Hansen’s Superior Predictive Ability test, or the Deflated Sharpe Ratio. Where feasible, the Probability of Backtest Overfitting should be estimated via Combinatorially Symmetric Cross-Validation. Out-of-sample results must be reported alongside in-sample results, with the out-of-sample period defined prior to optimisation.

• Sample-period justification: The start date of the backtest must be chosen by an out-of-sample test that compares expanding windows with rolling ones, and the choice must be disclosed, because more history helps only if it describes the market being traded. The strategy’s mechanism must be stated, together with the market feature it needs, whether that’s a cost that keeps others out, a participant who trades for reasons other than price, a session boundary, or a hedging flow, and that feature’s own history must be checked.

• Regime and structural-change analysis: Performance must be decomposed by market regime, with explicit identification of the conditions under which the strategy is expected to underperform. Regimes that come and go are only part of the problem, because some changes to a market happen once and stay. The backtest must carry a dated register for the traded market (venue, session hours, tick size, contract size, settlement, commission regime, circuit breakers and price limits, and overlying products such as weekly and daily options), each row with an effective date and a source. The backtest must be split at those dates, with each era’s edge reported with its own standard error. The behaviour the strategy feeds on (volatility, autocorrelation, the overnight share of the range, gaps, first-hour highs and lows, the volume curve) must be computed by year and tested for breaks, for example with the method of Bai and Perron (1998, 2003). A strategy whose results come from one era should be treated as a fact about that era. And because “the market changed” can excuse any bad run, that claim must be specific: name the change, date it, and show that performance breaks there and not somewhere else. Maximum drawdown and drawdown duration must be reported and contextualised.

• Risk management and portfolio context: The strategy must be presented with a defined position sizing methodology, explicit risk limits, and an analysis of how it interacts with other portfolio components. Standalone signal evaluation without risk management context is insufficient. The longest available history should be used to size risk and to stress-test, since it holds the only record of the worst cases, even where it isn’t trusted to measure the size of the edge.

• Robustness validation: The strategy must be validated using walk-forward analysis with fully disclosed window configurations, fitness functions, and the number of configurations tested. The walk-forward results must be presented as a necessary filter rather than sufficient proof of robustness. Walk-forward windows that cross a dated structural break must report each era separately, because a pass or a failure is evidence about the era it occurred in. If Monte Carlo analysis or synthetic data testing is employed, the assumptions underlying the method must be stated and their applicability to the specific strategy class must be justified. Claims of robustness derived from trade-shuffling Monte Carlo applied to strategies with dynamic position sizing, portfolio-level risk filters, or pattern-dependent entries should be treated with particular scepticism, as described in Section 6.

• Capacity assessment: The backtest must include an estimate of the maximum notional the strategy can deploy before its edge is degraded by market impact. This requires, at minimum, comparing typical order sizes to historical volume and depth-of-book data for the traded instruments. A strategy that is profitable at one contract but whose returns collapse at ten contracts has a capacity constraint that must be disclosed, and that is, for practical purposes, the binding constraint on the strategy’s economic value. Market impact is both temporary (price recovers after the order is absorbed) and permanent (the information content of the order shifts the equilibrium price), and strategies with high turnover or concentrated execution windows are disproportionately affected. The absence of any capacity estimate renders a backtest incomplete: the strategy may be valid but untradeable at the scale required to justify the infrastructure investment.

• Process discipline: The strategy creation and testing process should follow a defined, repeatable pipeline that enforces a scientific and statistically valid methodology. This pipeline should specify the sequence of steps from hypothesis formulation through data preparation, in-sample fitting, out-of-sample validation, and robustness testing, with each step’s criteria defined in advance rather than adjusted after the fact. The robustness tests and evaluation metrics employed must be appropriate to the specific class of strategy being tested and matched to what the researcher is trying to prove or disprove. A trend-following strategy, for example, demands different robustness criteria than a mean-reversion strategy, and a strategy intended for a single instrument requires different validation than one designed for a diversified portfolio. The absence of a structured evaluation process is itself a red flag: if the researcher cannot describe the pipeline that produced their results, the results should be treated with corresponding scepticism.