Three Traps in Backtesting: Survivorship Bias, Look-Ahead Bias, Overfitting

Between a backtest that runs and a backtest you can believe sit three classes of systematic error. None of them raises an exception. None of them breaks the equity curve. On the contrary — every one of them makes the result prettier. That is exactly why they are so hard to catch in your own work.

1. Survivorship bias: testing only on the companies that made it

Suppose you backtest a strategy from 2015 to today using the current S&P 500 constituent list. Every company on that list has already passed an extraordinarily strong filter: it is still in the index in 2026. The bankrupt, the acquired, the ones dropped for shrinking market cap — none of them are on the list.

So what you actually tested is not "a strategy on US equities" but "a strategy on companies already known to have succeeded." The gap between those two is not a few percentage points. Delisting is common in US markets: large numbers of names leave through mergers, take-privates, bankruptcies and listing-standard failures. Excluding them removes in advance precisely the part of the sample the strategy most needs to be tested against — the part the stop-loss rule exists to handle.

How to avoid it: the universe must contain symbols that existed during the test window but are delisted today, using their real price series through the delisting rather than truncating the series and treating it as a normal end. Our symbol library keeps delisted names and applies no survivorship filter; the market data API labels a symbol's delisted status explicitly in its response.

A counter-intuitive corollary: survivorship bias usually inflates the buy-and-hold benchmark more than it inflates the strategy. So on a biased universe, the strategy's excess return is systematically understated — the bias contaminates both sides, just unevenly.

2. Look-ahead bias: using information that did not exist yet

Look-ahead bias is far better hidden than survivorship bias, because it is usually off by exactly one bar.

The most common off-by-one: computing today's signal from today's data. Take "enter on a 22-day breakout." If the 22-day high window includes today, then "today's close ≥ the 22-day high" is nearly always true — today's high is already inside the window. The correct window ends yesterday:

hh = rolling_max(high, 22)
prev_hh = np.roll(hh, 1)      # shift by one bar: window ends yesterday
prev_hh[0] = np.nan
entry = (close > ma) & (close >= prev_hh)

That single bar is the difference between a worthless rule and a flawless equity curve.

Other common forms:

• Deciding on the close and filling at the close. Strictly, you cannot know the closing price until the close, so you cannot trade at it. The acceptable approximation is to decide on the close and fill at the next open; if you insist on filling at the close, at least leave room for it in your slippage assumption.
• Retroactive adjustment factors. Back-adjusted series are rewritten in full after every dividend and split. Backtesting a three-year-old strategy on a series downloaded today means using adjustment information that did not exist three years ago.
• Data revisions. Fundamentals, volume, and even some exchanges' daily bars can be corrected after the fact. Your backtest uses the corrected version; live trading at the time saw the uncorrected one.
• Index membership effective dates. Additions to the S&P 500 are announced first and effective later. Assuming knowledge before the effective date hands the announcement's price effect to your strategy for free.

How to self-check: for every indicator, ask "at the instant this is computed, did this number exist in the real world?" That one question catches ninety percent of look-ahead bias.

3. Overfitting: mistaking noise for structure

This is the hardest of the three, because it is not a bug — it is a process problem in which every individual step looks reasonable.

The chandelier strategy has six tunable parameters. Suppose you search a modest grid:

sma_period      6 values
breakout_period 5 values
atr_period      3 values
entry_mult      6 values
exit_mult       5 values
adx_threshold   4 values
--------------------------
combinations = 6 × 5 × 3 × 6 × 5 × 4 = 10800

One of those ten thousand combinations will be the best. The problem is this: even if the underlying data were a pure random walk, the best combination would still look rather good. The more combinations you search, the larger the share of the "optimal" result attributable to luck. When you report the return of the best combination, you are reporting a selected extremum, not an expectation.

The more insidious version is manual overfitting: you never ran a grid search, you just looked at a result you disliked, changed the SMA from 100 to 120 — a bit better — then to 150 — better still. That is not meaningfully different from an automated search, except that you kept no record and therefore do not even know how many combinations you tried.

Four practices that genuinely reduce overfitting:

1. Hold out a true out-of-sample window. Fit parameters on the first 70% of the data and run the last 30% exactly once. If out-of-sample performance drops sharply, what you selected was noise. "Exactly once" is literal — going back to tune after seeing the out-of-sample result converts it into in-sample data.
2. Want a plateau, not a spike. Always inspect the neighbourhood: if sma_period = 100 gives 24% annualised while 90 and 110 give 8%, that 24% is luck. Real structure shows up as a broad, gently sloping plateau in parameter space, never as a needle.
3. Remove parameters rather than adding them. Every filter you add (ADX, volume, day-of-week) improves in-sample results and degrades out-of-sample results faster. The bar for admitting a parameter should be "it has a defensible market mechanism," not "the curve looks better with it."
4. Validate across symbols. Run the same parameters on twenty names and check whether the conclusion is consistent. A parameter set that works on exactly one symbol describes that symbol's history, not the strategy.

4. Several other things that quietly distort conclusions

• Transaction costs and slippage. A strategy returning 15% annualised that trades 200 times a year gives up 20 percentage points to a one-way cost of 0.1%. For high-turnover strategies, cost is a core parameter, not an afterthought.
• Mixing data conventions. Computing breakouts from a single venue's closes while filtering on consolidated volume manufactures signals that do not exist in reality. Measured closing-price divergence between the two conventions reaches 7% — see Consolidated vs Single-Venue Volume.
• Windows that are too short. A history covering only a one-way advance cannot demonstrate the value of any risk rule. Our consolidated daily bars start 2024-07-01, an objective boundary of the data source; conclusions drawn on that window apply to that window and no further. Calling it "five years of validation" would be dishonest.
• The strategy itself is a survivor. You are studying trend following today because it was written into a great many books over the past decades. The strategies studied just as carefully that turned out not to work never reached you. This layer of bias cannot be corrected with data — only discounted by knowing it is there.

Closing: an uncomfortable standard

There is one crude test for whether a backtest conclusion deserves belief: write down the total number of attempts you made to reach it. If you cannot, or if writing it down reveals several hundred, then the conclusion's confidence is far below what you assumed.

The value of backtesting is not in finding the best parameters — it is in eliminating ideas that clearly do not work. As a falsification tool it is excellent. As an optimisation tool it will almost certainly deceive you.