Your backtest is lying to you

This is the story of one strategy that lied to us twice — once when it looked great, and once when it looked dead. It's the best illustration we have of why a single backtest number, good or bad, is never the answer.

Act one: the validated strategy

A momentum-continuation pattern — stocks gapping on catalysts, entered intraday. The backtest said profit factor 4.05. It cleared our review at the time and began firing live.

Its live signals — 677 of them — scored a profit factor of 0.48 when measured on minute-resolution data. It lost money at nearly the mirror-image rate at which the backtest said it would make money.

We killed it and wrote "classic overfit" in the changelog.

The detail that should scare you

Here's the part most post-mortems skip. When we re-simulated those same 677 picks on daily bars, the simulation said PF 1.50 — profitable. On 1-minute bars, the same picks scored PF 0.48.

Same strategy. Same entries. Same exits on paper. The only difference was the resolution of the price data — and it flipped the verdict from "works" to "loses." Intraday gappers fade fast; a daily-bar simulation simply cannot see the stop-outs that happen at 10:15am. Any backtest of an intraday strategy computed on daily bars is fiction, and a lot of published "verified backtests" are exactly that.

Act two: the autopsy that reopened the case

Two months later we noticed something uncomfortable: several of the biggest winners crossing our platform were names this dead strategy had flagged. So we re-investigated — full 12-dimension protocol, 456 deduplicated signals, tested against a 23,878-stock-day baseline.

The re-audit found our own kill diagnosis was partly wrong:

• The exit rule we'd tested it with was strangling it. With the tight stop/target, PF 1.54. With a plain 20-day hold, PF 2.16 — the profit target was amputating the right tail, and the right tail was the whole point.
• The signal carried real residual alpha after stripping out market, sector, and size exposure: +2.7% per 10 days (t = 3.09). Not just beta in a costume.

So we un-killed it, right?

No. Because it still fails the gates:

• The canonical permutation test against the broad universe: p = 0.056 and p = 0.052 at the two horizons. Borderline is a fail. A gate you bend once isn't a gate.
• The sample contains 2 bear-market signals. Two. We have no idea what this does when the tide goes out, and "it worked in a bull market" describes every bad strategy ever sold.
• Drop the top 5% of winners and the alpha collapses to +0.1% (p = 0.88). The entire edge lives in a handful of tail outcomes. The median trade is market-like.
• n = 456, below our 500-trade floor for promotion.

The verdict

Still killed as a strategy. The signal now feeds our scoring model as one input among many — a context where a tail-dependent, regime-untested signal can contribute without being trusted to trade on its own. If bear-market data accrues and the sample crosses 500, it earns a fresh trial. Not before.

What to take from it

1. A backtest is only as honest as its price resolution. Daily bars on an intraday strategy flipped PF 1.50 into PF 0.48 on identical trades.
2. The exit is part of the strategy. The same entries ranged from PF 1.54 to PF 2.16 depending on exit rule alone. Anyone quoting one backtest number tested one arbitrary combination.
3. Tail-dependence is a disclosure, not a footnote. If removing the top 5% of trades kills the edge, you are not buying a strategy — you are buying a lottery position with good branding.
4. Verdicts should survive their own autopsy. We re-opened our own kill, found our reasoning partly wrong, published that too — and the conclusion still held. That's what a record is for.

We run this gauntlet on everything we build. 24 strategies killed and published so far. The method is the product.