My hardest task was my grader's mistake
I built a science task from a fresh palaeomagnetism paper, and a frontier model scored 0.26. Most of that low score was my grader. Here is how I found out, and what I build first now.
I built a science task from a fresh palaeomagnetism paper, and a frontier model scored 0.26. Most of that low score was my grader. Here is how I found out, and what I build first now.
I built a science task that made a frontier model look bad, and it took me a week to find out the bad part was mostly my grader.
This is the story of that task, the numbers at each step, and what I now do differently. If you build evals or RL environments, the lesson costs you nothing to borrow. It cost me a week.
A model can't have memorized a paper that came out after its training cutoff. So take a fresh paper whose raw data is public, and ask the model to redo the analysis from that data. Now you are testing research skill, not recall.
Paper in, kept or killed task out.
I picked a palaeomagnetism paper published in October 2026: Chi, Sahroni, Hahn and Lhuillier, Geophysical Journal International, doi:10.1093/gji/ggag458. They measured how much Earth's magnetic field wobbled during the early Miocene, using the magnetisation locked into lava flows in the Rhön mountains in Germany. Their raw lab measurements are public in the MagIC database (contribution 20687, CC BY 4.0).
The headline number is a dispersion called S_B: how far the field's direction scattered between lava flows. The paper reports 37.5°. Getting there takes a long chain of expert judgment calls. Which part of each sample's demagnetisation curve do you fit? Which samples do you throw out, which flows do you merge, and which sites are good enough to keep?
The model in every run was Claude Opus 5.5, whose training cutoff is June 2026, at maximum reasoning effort.
The first version of the task told the agent the authors' choices: the thresholds, the rejection rules, the grouping. Opus scored 0.90. That number measured whether it could follow a recipe, not whether it could do the science.
So I rewrote the instruction to hold only the goal and the context a real expert would have had. That means the data, how the samples were collected, and what to report. Every choice the authors made on their own came out. Answer sources were blocked, and every run was checked for leaks.
| What the agent gets | Why |
|---|---|
| The raw lab measurements and the field notes | The expert had them |
| The goal: measure the field's scatter | The expert had it |
| Its own knowledge of published methods | The expert had it |
| The authors' thresholds, flags and site groups | Removed: the expert made them, so they are the answer |
| The paper and its result tables | Blocked: they are the answer |
Same model, same data, only the instruction changed. The score dropped from 0.90 to about 0.26.
Here is every score the task produced, in order. Each dot is one run.
Opus 5.5 reward, 0 to 1. Teal: below 0.5.
The v2 runs looked like a great result: a frontier model failing a real research task, three times out of three. Its S_B came out at 32 to 34° against the paper's 37.5°.
I almost wrote that up. Then I asked a better question. Not "is the model wrong?" but "is 33° actually wrong?"
My v2 grader scored the agent against the paper's numbers. But the paper's numbers are one team's answer. Another competent team, making different defensible choices on the same raw data, could land somewhere else.
So I rebuilt the analysis many times with different published choices: three published rules for rejecting outlier samples inside a flow, four site-quality cutoffs, four ways of grouping flows. I did it on three sets of fits. The first is the authors' best samples. The second is the authors' fits with their flagged samples kept, and the third is fits a script made with no hand-picking at all.
The scatter S_B with no cutoff. Each band covers three published outlier rules, four site cutoffs and four groupings, on one set of fits.
Each bar is the range of S_B that defensible analyses produce. The authors' 37.5° sits inside all three. But defensible analyses also land at 34 or 35°. With automated fits, the range reaches down to 32.3°, and Opus's three answers sit right at that edge.
Rescored against the band built from the authors' fits, the same three v2 answers got 0.50 to 0.73. Against the band that also holds automated fits, they got 0.98 to 1.00. Which fits I let into the band decided whether the task was hard or trivial.
That's the uncomfortable part, and I want to say it plainly. Most of the low score was my grader's error, not the model's.
Something was still off, though, and it was not in the headline number. I rebuilt one run's code from its log and compared it with the authors sample by sample.
| What I found | Sites | What it means |
|---|---|---|
| Kept samples the authors had flagged as bad | 6 | Its outlier removal stopped at 20% of samples, and these sites had more bad samples than that |
| Fitted the whole demagnetisation curve on samples hit by lightning | 4 | The authors fitted only the clean high-field end. Lightning leaves a strong second magnetisation at low fields |
| Too few samples passed its own quality check | 2 | The sites dropped out |
Most of these cost the model whole sites. And the sites it lost were not random.
Transitional: a site whose virtual pole sits more than 45° from the geographic pole. Exact hypergeometric test, p = 0.0038.
Transitional sites record the field while it is flipping or swinging far from its usual direction, so they carry much of the scatter. 7 of the 11 sites that Opus lost in all three runs were transitional, against 24% of the authors' sites. The chance of that happening at random is about 0.4%. Lose the transitional sites and S_B comes out low, which is exactly what happened.
Guess what? None of this showed up in the score. Sites kept, records, and the pole all scored full marks in every run. Only S_B differed, and with the automated fits it stayed inside the band. A grader that checks only the summary number can't see an error that happens at the level of a single sample. You have to grade at the step where the error shows.
(This diagnosis comes from one replayed run, and the replay is approximate. It gives 75 sites and 32.7°, against the 76 sites and 32.4° that run reported. I have not checked the same causes in the other two runs.)
So I rebuilt my whole process to put the grader first. Before any agent sees a task, I write down the range of answers the field would accept, and the rule for keeping or killing the task. Then I run one blind probe.
Then I tested the rebuilt process from scratch, on the same paper, with a fresh agent doing the building. It reproduced the paper's main table from the raw data, built the band, wrote down the kill rule, and ran one blind probe of Opus 5.5. The probe scored 0.97. The rule said kill, so the task was killed.
Wall-clock hours. Grey: the paper gate. Light teal: building the band and the probe. Teal: the blind probe run.
The first version of this task took a week to reach the wrong conclusion. The rebuild reached the right one in 1.3 hours.
The second paper I put through the rebuilt process, an astronomy paper, died even faster. Its key numbers already sat in a public slide deck from an institute meeting, two and a half years before the paper went online. A model trained on the web could have read them. The gate caught it in six minutes.
Two papers in, two kills, both cheap. A third is in progress. That's the point of the process: most papers should die early, before they cost a week.
| Principle | What it means in practice |
|---|---|
| Build the band before the task | Write down the range of answers competent experts would defend, and the keep or kill rule, before any agent runs |
| The grader accepts what the field accepts | A method the field publishes and uses scores full marks, even if the paper's authors did it differently. Only what the field itself calls an error loses points |
| Grade at the step where the error shows | If a mistake hides in the summary number, check the step where it happens |
| Check coherence with the rest of the field | An answer that matches this dataset but breaks laws that hold elsewhere in the field fails |
| Leaks hide in talks and slides | Search meeting archives and slide decks, not only preprint servers |
| Kill cheap | A rule written down before the run decides. One cheap probe beats a week of building |
I'm putting together a small public set of sample tasks that you can try yourself. I'll link it here when it is up.