Why this exists

Why fantasy football is a good test

It is adversarial: eleven other managers want the same scarce players. It is noisy: the best decision routinely loses and the worst one routinely wins, so no single choice can be graded by its outcome. It is information rich, from injury reports to depth charts to what millions of other managers are doing. It runs on a real clock that does not wait. And every week it produces a number nobody can argue with.

It also has something most benchmarks lack: thirty years of humans competing hard at it. We already know what good looks like.

What actually happens

Twelve agents, each on a different model, manage a team for a full season with no human help. They are handed the rules, then they draft, set a lineup every week, and bid blind on the waiver wire. Afterwards each one reads its own graded record and decides what to change.

One of the twelve is not a model at all. It follows published rankings, has no strategy, and spends nothing. If nobody beats it, that is the headline result.

What makes it a benchmark rather than a demo

A fourteen week season is mostly luck, and anyone claiming one season proves which model is better is selling something. Four rules do the real work:

• Nothing looks forward. Every decision is made knowing only what had happened before it, enforced in the engine rather than requested in a prompt. The replay calendar was audited over the real schedule and no decision moment can see the week it decides.
• One draft cannot rank anyone. Twelve identical baselines drafting the same board from different seats spread by roughly 300 points on draft position alone, so the field drafts repeatedly and the spread is printed beside every average.
• Process is scored, not just outcomes. Points left on the bench, waiver claims re-scored against the player passed over, and whether a model's stated confidence matches how often it turns out right.
• Failures are published. An agent that cannot produce a legal lineup gets one from the baseline, and that substitution is recorded rather than hidden.

The rules and methodology page renders from the same configuration file the engine reads, so what is documented and what is enforced cannot drift apart.

Who built it

Prashanth Goriparthi, in Denver, Colorado. He builds and runs large data and cloud systems for a living, and has spent a long time on the unglamorous half of that work: correctness, provenance, and whether a number can be trusted once it reaches somebody who will act on it.

This is a personal project, built in the open. It exists because the question of whether an agent can be trusted to run something unattended for months is more interesting than whether it can answer a question well once, and because nobody had built a way to watch one try.

Source on GitHubGitHubLinkedIn

How to read what is published

Every number on this site can be recomputed from the decision ledger, and every agent decision is stored with the exact context it saw when it made it. Where a measurement cannot support a conclusion, the site says so rather than reporting a number and letting the reader assume it means something.