How frontier agents fail a science benchmark
I read 618 failing runs of five frontier agents on Terminal-Bench-Science. They rarely break. They get the science wrong, and most of them say they are done.
I read 618 failing runs of five frontier agents on Terminal-Bench-Science. They rarely break. They get the science wrong, and most of them say they are done.
I read through every failing run of five frontier AI agents on a hard science benchmark, 618 runs in all, to find out how they fail.
The benchmark is Terminal-Bench-Science v0.1.0. It has 70 research tasks across the sciences. Each one is a real workflow that an agent carries out in a terminal. Its public leaderboard publishes every run. That makes it one of the few places where you can study failure at scale instead of guessing at it.
The short version: the agents rarely break. They do the science wrong, and then they tell you they're done.
| Benchmark | Terminal-Bench-Science v0.1.0, 70 tasks (doi:10.5281/zenodo.22110254) |
| Agents | Opus 5, Opus 5.5 and Fable 5 in Claude Code; GPT-5.6 Sol and GPT-6 Astra in Codex. Maximum or xhigh effort, 8-hour limit |
| Runs | 1,050: three per task and agent, from the public leaderboard |
| Failing runs labelled | 618 |
| How | A reviewer agent read each run against a written codebook. A second reviewer relabelled 75 runs without seeing the first labels |
A failing run can be the agent's fault or the task's. So the first question was simple. Who failed here?
618 failing runs, five agents, 70 tasks. First reviewer's labels.
In 523 of the 618 failing runs (85%), the reviewer found the agent at fault, not the task. And 467 of those 523 were errors in the science itself. Crashes, time-outs, format slips and quitting early add up to only 56.
The second reviewer was stricter here: on the 75 runs it relabelled, it called 72% agent failures where the first reviewer called 79%. So read 85% as an upper end.
Which raises the better question. When an agent gets the science wrong, is it using the wrong method, or the right method badly?
467 science errors. Teal: the right method with too little accuracy. Grey: a wrong method, model or result.
About half and half. 246 runs used the right family of methods with too little accuracy. 221 used a wrong method, model or convention, or reported a result that is physically impossible. The two reviewers agreed on this split only about half the time, so treat it as rough.
This one stuck with me. In 454 of the 523 agent failures (87%), the agent's final message says or implies that the task is complete.
Dot: the share of agent failures that end by claiming the task is done. Line: 95% interval, resampling tasks.
The two strongest agents claim success less often: 71% of their failures, against 91% for the other three. But they also fail fewer and harder tasks, so this is not a like-for-like comparison.
There's a hole in this number, and I want to name it. Agent harnesses usually close with a summary of what they did. I have not labelled the passing runs the same way. So I can't yet say how much of the 87% is overconfidence, and how much is the harness's normal goodbye.
For each agent failure, the reviewer named the specific mistake. Of 173 named mistakes, 58 show up in models from both Anthropic and OpenAI, and those 58 cover 336 of the 523 agent failures.
173 named mistakes. Teal: made by two or more of the five agents.
That sounds like a shared blind spot. It might be. But it comes with real confounds:
If a mistake is that common, you should be able to build new tasks around it. So I tried with one named mistake on one task. Opus 5.5 failed the original task in all 3 of my reruns. Then I built three smaller tasks around the same mistake.
Opus 5.5 in Claude Code, maximum effort. Each mark is one run.
It passed all 5 runs of the new tasks. Isolating the trap made it easy. My guess is that the original fails because the mistake is buried among many interacting effects, but I haven't tested that.
That result changed my approach. Instead of rebuilding traps on simulated data, I now build tasks from fresh papers and their real raw data. The story of the first one is its own lesson.
| Field | Agreement between reviewers | Cohen's κ |
|---|---|---|
| Who failed (agent, task, unclear) | 77% | 0.42 |
| Science or generic error | 100% | 1.0 |
| Claimed success | 93% | 0.42 |
| Near miss | 73% | 0.38 |
I'm putting together a small public set of sample tasks built the same way, so you can try one yourself. I'll link it here when it is up.