Why market data is harder than you think

Overview

There are two kinds of traders: those who know data is hard, and those who are yet to find out.

On face value, getting data looks easy: download prices and find an edge.

A quant shop is more likely to budget for rigorous data management and make someone responsible for it. Retail traders tend to choose a provider, get comfortable with its conventions, and build years of research around them. Reconsidering that choice gets progressively harder.

The examples here come from Marketfeed, the data archive behind our trading platform, and include vendor conventions and mistakes in our own software.

A standard nomenclature

If you trade E-mini S&P 500 futures, ES probably seems like a sufficient name. Your charting package understands it, and so does everyone you discuss it with. Once you start moving data and orders between systems, however, you need to be considerably more specific. A ticker belongs to a namespace, and each provider gets to decide what its names mean.

We use class:venue:ticker as our standard nomenclature. Thus fut:XCME:ES identifies a futures root on CME, using the Market Identifier Code (MIC) XCME for the venue. IBKR uses CME as its exchange label. That small difference is already a translation you have to get right.¹

Including the asset class and venue lets us distinguish products that happen to share a ticker. An index and a future might have colliding names; the same ticker on two exchanges might identify different securities. You also need a way of specifying a particular futures expiry. It’s easy to postpone these decisions when your universe consists of a handful of instruments, but changing identifiers after they’ve spread through strategy configurations and historical position records is an unpleasant job.

Then consider the vendors. Databento, FirstRate, and TradeStation each have their own naming conventions. IBKR has several within the one integration: its underlying search symbol, trading class, and contract ID identify different things. The search might return a family of contracts from which you still have to select the product and expiry you want.

Micro Silver is a useful example:

For reliable support of multiple vendors, you need a common internal identity and an explicit mapping to each vendor’s namespace. I consider this a basic design requirement. You can represent that identity as a string, an object, or a registry key; the system must resolve an instrument using the vendor’s namespace and the required contract details, or report the ambiguity. Adopting a supplier’s ticker throughout your strategy code just makes that supplier’s assumptions harder to remove later.

The venue and supplier need separate meanings too. Buying CME history from another supplier shouldn’t change the instrument’s identity.

Even class:venue:ticker needs a historical interpretation. A security can change its ticker, and an unrelated security can later inherit the old one. Including the venue won’t resolve reuse on the same exchange. You need a persistent instrument identity behind the name, with mappings recording when each name was valid. Today’s symbol lookup is insufficient for yesterday’s backtest.

Order generation and order management

A common nomenclature also lets you decouple order generation from the backend order management system (OMS). The strategy expresses what it wants to trade using the canonical identity. The OMS keeps its orders and positions under that identity, while a broker adapter translates the outgoing instruction into the contract representation the broker accepts.

That gives you this round trip:

Order: strategy → order for a real canonical contract → OMS → broker adapter → broker.

Fill: broker execution → broker adapter → canonical contract → OMS position and strategy state.

This confines a new broker’s naming conventions to its adapter. Your strategies and position book continue to use the same instrument identities. There will still be work to do on contract sizes, supported order behaviour, and the new broker’s peculiarities, but that work can stay out of the strategy’s trading logic.

The return path deserves as much attention as the outgoing order. A fill has to update the correct position, including the correct expiry. Keeping the broker’s identifiers in execution records is useful for reconciliation, but you also need to translate the execution back into the identity the rest of your system understands. Otherwise, two integrations can leave you with two apparent positions in what is actually the same instrument.

Our Micro Silver mapping once used SI as its alias, the same as standard Silver. A search on that underlying can return both products. The micro needs the SIL trading class to distinguish it, and failing to make that distinction left both contract selection and reverse lookup vulnerable to error. Standard Silver is 5,000 troy ounces; Micro Silver is 1,000. Routing a one-contract Micro Silver order into standard Silver would give you five times the intended exposure.^{2 3}

So an ambiguous mapping must fail. The same applies to a missing broker route or an expiry that isn’t listed. If an execution comes back with an unrecognised identity, it needs investigation. Guessing is a poor policy for either placing trades or keeping the books.

Here, fut:XCME:ES identifies the root. The backtester can run a strategy on a synthetic continuous series and generate signals from it. The strategy’s orders must still name a real contract, such as fut:XCME:ES:U26 for September 2026. Selecting that contract requires a roll rule. Our archive supplies the dated contracts, the trading engine determines the rolls, and the separate OMS manages orders. Marketfeed doesn’t submit them.

Timestamps

Suppose your one-minute bars are labelled with their opening time. The row marked 10:00 contains the high, low, and close from the minute that follows. If your backtester makes that completed row available at 10:00, it has given the strategy a minute of future information.

Databento uses opening timestamps for its OHLCV bars. Our legacy archive used closing timestamps, so moving between them required shifting each timestamp by the interval.^{4 5}

Vendor-local timestamps for a European instrument can turn out to be in New York time. This seems implausible, but this is what some data vendors (such as FirstRate Data do!). And just because data is relative to UTC doesn’t guarantee accuracy. A fixed UTC offset may agree for months, then fail at daylight saving. Worse still, the US and Europe change on different dates. Those intervening weeks are particularly useful when comparing imports from different providers.

For slower data, the delay between observation and availability can be much larger. Commitments of Traders reports generally describe positions on Tuesday, with publication on Friday at 3:30 pm Eastern. Put those figures against Tuesday in a backtest and the strategy gets several days’ notice of information it couldn’t have had. Even a rule that always moves them to Friday will be wrong around some delayed releases. The CFTC’s release schedule includes the exceptions.⁶

Macroeconomic observations can be revised repeatedly after publication. Today’s download may contain a later estimate for an old quarter. Even delaying that figure until the original release date gives the strategy a version that didn’t exist then.

FRED’s default history gives you what is known today about the past. Its historical real-time periods and ALFRED’s vintages allow you to retrieve earlier versions. For intraday trading, you may need to go further and establish when the release reached the service you actually used; FRED distinguishes the original release date from availability on its own services.⁷

Look-ahead bias introduced here can survive a very elaborate validation process. Walk-forward testing will repeatedly give the strategy data before it was available, just as the initial backtest did.

What do you mean by a daily bar?

Ask two providers for daily data and you can get different bars even when they agree about the underlying trades. An overnight futures session crosses midnight, so grouping by calendar date and grouping by exchange trade date produce different results. Databento’s supplied daily OHLCV uses UTC dates. If you want exchange-session bars, you need to aggregate a finer interval over the appropriate window.⁸

Providers handle holiday half-sessions differently: reporting them separately, folding activity into another trade date, or omitting the date. That changes both the rows present and their open and range. Your previous-day-high signal inherits those choices; a session filter can’t regroup trades already compressed into daily bars.

And what does “close” mean? It could be the last electronic trade, an auction price, or the exchange settlement. Settlement is used in valuation and clearing, and it needn’t equal the final trade in your chosen session. These are all useful prices, provided you know which one you’re using.

If your roll rule uses volume to select a contract, the choice of series can change which contract wins. The publication time matters as well: a decision based on yesterday’s official volume has to wait until the figure is available.

Session hours change

Nasdaq has announced 23-hour trading, five days a week, from 6 December 2026; NYSE Arca plans the same start, subject to some conditions. Nasdaq adds a 9:00 pm–4:00 am Eastern overnight session, with a daily pause from 8:00 pm to 9:00 pm. The regular 9:30 am–4:00 pm session and closing auction remain.⁹

The Sunday launch starts the trading day labelled Monday, 7 December. Nasdaq assigns trades from 9:00 pm to midnight to the next trading date; those from midnight to 8:00 pm carry the current date.¹⁰ Midnight in New York doesn’t start a new trading day. Consider these timestamps in the launch week:

Monday’s session spans two New York calendar dates and two UTC dates. In Melbourne, it runs from 1:00 pm Monday to noon Tuesday AEDT. That conversion changes with daylight saving; a fixed offset won’t work all year.

Grouping by New York calendar date would split sessions and combine parts of different trading days. Grouping by UTC date would put Monday’s last hour into Tuesday’s bucket. Keep the execution timestamp for sequencing events and the exchange trading date for session membership. Changing the date label mustn’t change when a trade actually happened.

Vendors can supply core-session bars, full-session bars, or calendar-day aggregates. An opening-gap signal inherits that choice. A full-session open comes from the first eligible overnight trade; the official closing auction remains at 4:00 pm. And a completed full-session bar labelled Monday isn’t available until Monday’s 8:00 pm finish. Give it to a strategy at the regular close and you’ve supplied four hours of future information.

Session boundaries need separate treatment too. Nasdaq cancels outstanding overnight orders at 4:00 am, although the trading date continues. A backtest must account for that cancellation when deciding whether a later trade could fill an order.¹¹

Nasdaq’s proposed handling of certain corporate actions would halt affected securities before 9:00 pm on the evening preceding their effective date, then reopen them at 8:00 am. A split adjustment keyed blindly to your computer’s midnight can put observations on the wrong price basis.¹²

Use a dated exchange calendar: for a Monday exchange holiday, Nasdaq’s FAQ puts the reopening on Monday evening. Keep core-session and full-session definitions explicit, and build those bars from minute data if necessary. Version the rule change so December’s hours aren’t imposed on earlier history. Verify each provider’s definition and coverage before the cutover; an unchanged field called date tells you remarkably little.

Continuous futures prices

A continuous futures chart makes it easy to think of ES as one instrument with decades of history. In reality, somebody selected a succession of expiring contracts and joined them using a roll rule. They may then have adjusted the prices to remove the gaps between contracts.

For the same trade date, 10 July 2014, we had these three opening prices:

The actual session opened at 1967.50. FirstRate’s historical level had been shifted by 384.75 points; the Panama reconstruction carried a 626.00-point shift. The adjusted numbers include the effects of later contract transitions. Neither was the price at which that session opened.

With additive, or Panama, adjustment, earlier history is shifted to remove a gap at each roll. The cumulative shift depends on the roll rules and where you anchor the series. Our reconstruction used the end of the query window as its anchor, so a request ending in 2014 and one extending through 2026 could return different prices for the old session.

Adding a constant preserves point differences inside a uniformly shifted segment, but percentage returns change because the price in the denominator changes. Absolute price thresholds and price-based sizing can change as well. Ratio adjustment preserves proportional relationships within a uniformly scaled segment, while changing point distances.

A strategy can use synthetic prices to generate signals, provided its calculations make sense under that adjustment. Any resulting limit or stop level must be translated onto the real contract’s price scale. Simulated fills must use that contract’s market data too; the synthetic series never traded.

There were actual defects in the sample too. Our investigation found inconsistencies in FirstRate’s adjustment ladder and differences between its minute and daily roll construction. Those findings refer to the files we examined, and the figures above shouldn’t be read as an endorsement of each method.

I prefer to keep the dated contracts and make the roll and adjustment rules explicit, so there’s enough information to rebuild the series and explain a discrepancy. A pre-stitched download is convenient, but it commits you to the supplier’s decisions.

Equities are complicated too

Take a four-for-one split. The holder ends up with four times as many shares, with the corresponding price basis divided by four. Adjusting the chart makes comparisons across the event easier, but an execution simulation must also keep track of quantities and pending orders. Changing the price scale while leaving an order or position on the old basis produces an inconsistent account.

Dividends are another way to get apparently good results with remarkably little effort. A simulation using raw prices can credit the dividend as cash. A total-return series already incorporates distributions into its return measure. Feed an already dividend-adjusted return stream into a process that adds the dividends again and you can count the same benefit twice. It’s an easy mistake if the adjustment policy is hidden somewhere inside the platform.

Tickers change, and get reused

Meta Platforms changed its Nasdaq ticker from FB to META on 9 June 2022. Its Class A shares retained the same CUSIP. The stock’s history still extends back through its time as FB. Earlier that year, however, META identified Roundhill’s metaverse ETF on NYSE Arca, which changed to METV on 31 January. The bare ticker has belonged to two quite different investments.¹³

Treat the Meta rename as the birth of a new security and you lose valid history. Join everything called META and you can splice an ETF’s prices into a technology company’s record. Returns across that join are meaningless, and corporate actions can end up applied to the wrong security. A chart can look quite plausible while the instrument underneath it has changed.

This is why the reference data needs to remember which security used each name, on which venue, and when. A routine rename should preserve the connection to existing positions and orders; a new security needs its own identity. Our archive work required separate event ledgers and rename operations that kept bars, corporate actions, and instrument records connected. Changing the catalogue label alone would have stranded the old files.¹⁴

Today’s surviving tickers may have excellent histories and still be the wrong universe for a historical backtest. Ask how your provider handles delisted securities before spending much time on an equity model.

Missing data and data cleaning

A missing bar tells you very little by itself. The exchange might have been closed, the instrument might not have traded, or there might have been a failure somewhere between the exchange feed and your local file. How do you distinguish them?

Databento omits an OHLCV record when no trade occurs. We encountered a genuine one-minute aluminium series whose median spacing was two minutes, because enough minutes had no prints. A classifier that inferred the bar interval from the median gap got it wrong.^{15 16}

Elsewhere in the migration, the missing periods were entire years: BFX had no rows for 2024, COMEX aluminium ALI lacked 2018, and Henry Hub HH lacked 2011. If you summarise coverage using only the first and last timestamp, those holes disappear from the summary. We had to record the separately covered periods.¹⁷

Suppose a feed stops in the afternoon. The morning’s observations can still produce a daily bar that passes routine OHLC checks, although its volume may be understated and its final price far from the session close. Your strategy can make the wrong entry or roll decision without encountering an obviously malformed row.

Compare the exchange calendar and the vendor’s feed-condition reports with your own import logs. A holiday calendar can’t explain why your downloader stopped halfway through a Tuesday; a gap in your file doesn’t establish that the vendor lost anything.

Outlier filters need care too. In one Brent validation case, the front contract fell about 22% and a back month about 12%. Using the front month as the reference made the back-month move look suspicious, but nearby maturities corroborated it. Different parts of the futures curve can behave differently, especially during the sort of market you most need to understand before trading a strategy.¹⁸

It would be unfortunate to remove those observations because they make the model harder to fit. Yet that is effectively what a badly chosen cleaning rule can do. Forward-filling a gap is also a modelling decision: the repeated value is something you’ve supplied. Both operations need a reason, and you should retain a record of the changes so you can examine them later. A dataset that looks less untidy isn’t necessarily more accurate.

Combining vendors and resolutions

Long histories often come from several sources. The oldest period might contain minute bars from one vendor, with second bars from another covering the later years. Trade-level records might exist for only part of the range. A single start date and a description saying “20 years of data” don’t tell you much about those differences.

An overlap gives you the opportunity to compare suppliers, but you still need a rule for deciding which observations your reader serves. Concatenating the files can double-count the same period. Filling gaps can be more subtle: if the source changes inside a bar you’re constructing, you need to know whether the contributions can legitimately be combined. You can otherwise create a bar that neither vendor delivered.

Compare the same dated contract, session, and interval, with matching timestamps. Check source boundaries and daylight-saving weeks. Agreement in the overlap gives some confidence in the older source, but cannot verify history for which no comparison exists.

And be honest about the resolution. If an early period contains only one-minute OHLCV, a request for five-second bars should fail and identify the limiting period. Repeating or subdividing those observations creates prices for which you have no evidence. You can choose a coarser experiment or acquire finer data, but that should be a conscious decision by the researcher.

Keep track of sources and native resolutions even when you serve a common interval. Investigate a performance change at a vendor boundary before attributing it to a market regime. I’ve discussed why those resolution limits matter in The lie hidden inside daily bars .

Keeping the archive usable

You also have to keep the data you already acquired. During implementation review, we found a bug that could replace a yearly partition with only the newly fetched rows, silently discarding the rest. The fix merged old and new contents, kept recovery copies, and recorded replacement steps so an interruption could be recovered.¹⁹

Checking the delivered file, decoding it, mapping its instruments, and storing it safely all have to succeed. Coverage records must follow the durable writes; otherwise a subsequent download can skip the lost period because your own records claim you already have it.

A completed job can leave individual sources stale. Report which series advanced and which failed, and count newly acquired observations separately from rows already in the archive.

Our IBKR margin history depends on prospective captures. A missed day leaves a gap that today’s figure can’t repair; we’d need another source of historical evidence. Such information needs collecting before you decide to research a question that depends on it.²⁰

Keep a backtest you can replay

A backtest should be deterministic and replayable. Given the same preserved inputs, software, configuration, and initial state, it should generate the same orders, simulated fills, positions, and P&L. If it uses randomness, fix and record the seed. Control event ordering and the runtime too: a seed won’t cure inconsistent ordering in parallel processing.

Give each run an identifier and retain the data snapshot or recoverable version, including reference data. Record the strategy and engine versions, parameters, initial positions and cash, session and adjustment policies, and execution assumptions. Preserve the order and fill log, with timestamps and the observations behind each decision.

The first check is to replay the saved run against its frozen inputs. If the result changes, investigate your own software or environment. Next, download the same historical range again and compare the records using identical query and normalisation rules. Keep the old copy. A hash can flag a difference, but a record-level comparison tells you which bars or reference values changed; different compression or file ordering needn’t mean different market data.

Run the frozen strategy against the new data as a separate experiment. Now you can trace a changed trade back to the changed observation that caused it. Corrections to genuine errors are welcome, but you still need to know which previous research results they affect. An unchanged backtest doesn’t prove the history is unchanged either; the strategy may simply be insensitive to the revised rows.

Automate this audit after data updates to distinguish provider revisions from your own changes. Repeatability alone doesn’t establish that the data is correct.

What happens when you report an error?

My experience with TradeStation has been particularly poor. In 2020 I raised concerns with TradeStation relating to its data integrity. As a result of these issues, I stopped using TradeStation’s data. In 2024, I revisited the issue. Four years had gone by, but nothing had been done to address the data issues. When pushed, TradeStation acknowledged that the problem existed, but provided a defence: “This problem is not an easy fix.”²¹ Needless to say, I would not recommend any trader rely on TradeStation’s data.

Other providers have acknowledged and corrected basic errors when challenged. That is a better response, although such elementary mistakes raise questions about their quality assurance. The customer shouldn’t have to perform the vendor’s first useful checks.

Databento is a good example of a more open approach. It publishes data-integrity issues and exposes dataset conditions and modification dates through its API. You can identify dates with known degradation, follow an investigation, and decide whether affected research needs to be rerun.²²

The cost of changing providers

For many traders, the inertia required to change data providers is sufficiently large that they avoid doing it. This can impose significant restrictions on the type of strategies that can be developed and the markets that a trader will consider analysing. Often, the choice of data provider is not a conscious decision. It may be dictated by platform integration restrictions or perhaps just a case of using what everyone else seems to be using.

Taking data seriously

A suitable provider and platform can do much of this work for you, provided you understand their conventions well enough to judge the results. Ask them:

• Identity: What exact instrument does this observation or order represent? How are ticker changes and reuse handled, alongside vendor translations and returning fills?
• Construction: Which session, close, volume definition, roll rule and adjustment policy produced these values?
• Availability: Could the strategy have received this version of the observation at the simulated decision time?
• Coverage: Which periods and resolutions are actually present, and how are closures, inactivity and failures distinguished?
• Reproducibility: Can you replay the saved run exactly, and identify which observations changed after a fresh download?

Data Quality: The Contaminated Foundation goes into the broader catalogue of errors. The problem for a strategy developer is deciding which of them the chosen tools deal with, and which remain the developer’s responsibility.

Retail traders can adopt this discipline on a smaller scale. Keeping recoverable data and testing a second source early gives you a chance to find problems before the cost of moving becomes an excuse to tolerate them.