September 18, 2026
Twenty-four seasons, and what did not hold up
The backtest went from three seasons to 24, and half the rules we were proud of turned out to be noise.
The first version of the picks of the week was fitted on the 2023 to 2025 seasons. The lock rule went 59 and 6. The bold call went 34 and 32. We wrote rules on top of that: a bigger bump on tight agreement, a harder press on the lock, more Elo in the playoffs. Then we asked the obvious question: what happens on more seasons?
FiveThirtyEight's published QB-Elo file covers every game from 1999, and nflverse has closing spreads for the same games, 6,064 of them. Replaying the picks over all 24 seasons took an evening and rewrote the site.
- The lock rule held: 379 and 71, 84%, against an average forecast of 80%. Real, and still the site's best single pick.
- The bold call did not: 217 and 233, 48%, while Elo claimed 62% for its side. When Elo and the market are ten or more points apart, Elo's side wins half the time. The market is right. The bold call stayed on the site as the contrarian view with its record printed next to it.
- The tight-agreement bump shrank from eight points to two. The lock press from five to two. The playoff Elo weighting went away entirely. Each was a three-season pattern that a 24-season replay averaged out.
- The bye-week bonus survived; the 25-point rest adjustment was calibrated on the same 24 seasons.
The lesson is in the Nudges page, which lists every finding with its evidence and a live tally this season, and in the changelog, which keeps the reversions. A model that only shows what worked is a marketing page.