November 15, 2026·research

How frontier agents fail a science benchmark

I read 618 failing runs of five frontier agents on Terminal-Bench-Science. They rarely break. They get the science wrong, and most of them say they are done.

Vedant Patel·AI Researcher·5 min read

I read through every failing run of five frontier AI agents on a hard science benchmark, 618 runs in all, to find out how they fail.

The benchmark is Terminal-Bench-Science v0.1.0. It has 70 research tasks across the sciences. Each one is a real workflow that an agent carries out in a terminal. Its public leaderboard publishes every run. That makes it one of the few places where you can study failure at scale instead of guessing at it.

The short version: the agents rarely break. They do the science wrong, and then they tell you they're done.

What I looked at

BenchmarkTerminal-Bench-Science v0.1.0, 70 tasks (doi:10.5281/zenodo.22110254)
AgentsOpus 5, Opus 5.5 and Fable 5 in Claude Code; GPT-5.6 Sol and GPT-6 Astra in Codex. Maximum or xhigh effort, 8-hour limit
Runs1,050: three per task and agent, from the public leaderboard
Failing runs labelled618
HowA reviewer agent read each run against a written codebook. A second reviewer relabelled 75 runs without seeing the first labels

Agents rarely fail on mechanics

A failing run can be the agent's fault or the task's. So the first question was simple. Who failed here?

618 failing runs by cause

618 failing runs, five agents, 70 tasks. First reviewer's labels.

In 523 of the 618 failing runs (85%), the reviewer found the agent at fault, not the task. And 467 of those 523 were errors in the science itself. Crashes, time-outs, format slips and quitting early add up to only 56.

The second reviewer was stricter here: on the 75 runs it relabelled, it called 72% agent failures where the first reviewer called 79%. So read 85% as an upper end.

Half the science errors are close calls

Which raises the better question. When an agent gets the science wrong, is it using the wrong method, or the right method badly?

Science errors split into accuracy and method errors

467 science errors. Teal: the right method with too little accuracy. Grey: a wrong method, model or result.

About half and half. 246 runs used the right family of methods with too little accuracy. 221 used a wrong method, model or convention, or reported a result that is physically impossible. The two reviewers agreed on this split only about half the time, so treat it as rough.

Most failing runs end with "done"

This one stuck with me. In 454 of the 523 agent failures (87%), the agent's final message says or implies that the task is complete.

Claimed success by configuration

Dot: the share of agent failures that end by claiming the task is done. Line: 95% interval, resampling tasks.

The two strongest agents claim success less often: 71% of their failures, against 91% for the other three. But they also fail fewer and harder tasks, so this is not a like-for-like comparison.

There's a hole in this number, and I want to name it. Agent harnesses usually close with a summary of what they did. I have not labelled the passing runs the same way. So I can't yet say how much of the 87% is overconfidence, and how much is the harness's normal goodbye.

Different labs, same mistakes

For each agent failure, the reviewer named the specific mistake. Of 173 named mistakes, 58 show up in models from both Anthropic and OpenAI, and those 58 cover 336 of the 523 agent failures.

Named mistakes by how many configurations made them

173 named mistakes. Teal: made by two or more of the five agents.

That sounds like a shared blind spot. It might be. But it comes with real confounds:

  • One reviewer named all the mistakes on a task, so names converge by design.
  • A task with one designed trap makes every failing agent fail the same way.
  • Lab and harness are the same split here, Claude Code against Codex.

A pilot: one trap did not reproduce one failure

If a mistake is that common, you should be able to build new tasks around it. So I tried with one named mistake on one task. Opus 5.5 failed the original task in all 3 of my reruns. Then I built three smaller tasks around the same mistake.

Pilot: original task failed 3 of 3, new tasks passed 5 of 5

Opus 5.5 in Claude Code, maximum effort. Each mark is one run.

It passed all 5 runs of the new tasks. Isolating the trap made it easy. My guess is that the original fails because the mistake is buried among many interacting effects, but I haven't tested that.

That result changed my approach. Instead of rebuilding traps on simulated data, I now build tasks from fresh papers and their real raw data. The story of the first one is its own lesson.

How far to trust this

FieldAgreement between reviewersCohen's κ
Who failed (agent, task, unclear)77%0.42
Science or generic error100%1.0
Claimed success93%0.42
Near miss73%0.38
  • Both reviewers are Claude models, the same family as three of the five agents. This measures consistency between agents, not agreement with a domain expert.
  • Tasks only enter the benchmark if they are hard, so a high science-error share is partly built in.
  • This covers five of the twelve leaderboard configurations, each in its own lab's harness.
  • It is not a ranking. Failure shares depend on which tasks each agent fails.

I'm putting together a small public set of sample tasks built the same way, so you can try one yourself. I'll link it here when it is up.