Roast Bench asks whether a model can roast coffee. Each item gives it a green coffee and a roaster, and nothing else: no batch size, no target curve, no reference roast. The model chooses the batch and the charge temperature and writes one program in the Roast DSL. We roast that program on a simulator fitted to that coffee on that machine, and grade the curve it makes.
The task
- 1,223 items on 15 roasters: electric drums, air roasters and gas drums, from sample roasters to production machines.
- The roaster is described by the channels it can move (heat, air, drum speed), their ranges, its capacity and the longest roast it allows.
- The coffee is described the way its bag would be: country, region, farm, producer, variety, process, altitude, harvest, and density and moisture when they're known.
- The program has to compile and be legal on its machine: the right channels, values in range, steps in order, a drop inside the time limit, a batch that fits. An answer that isn't legal scores zero.
The grading
The rubric uses the markers from roast-quality research that a simulated curve can actually show. Nine-tenths of the score is the core:
- Timing. Time to first crack, and the total roast time, inside the range for the roaster's class.
- Too fast. A start under the class's floor, or one likely to scorch.
- Momentum into crack. Enough rate of rise over the last minute before first crack.
- Stalls. The rate of rise collapsing or going negative before crack.
The last tenth is smaller markers for the machine's class (gas, air or electric drum) and for natural and fermented lots. A roast that never reaches first crack scores zero.
Roast degree and development aren't scored yet. A simulator has no colour reading to trust, and development can't be judged without knowing what the roaster meant to do. Both belong to a second section, where the model first declares the roast it's aiming for, which isn't built yet.
The human lines
The same rubric, run on recorded roasts:
- The average roaster: a median of 96.4, over 67,019 community roasts of 49,925 coffees.
- Experts: a median of 97.7, over 26,140 roasts of 964 coffees by verified professionals and championship entrants.
First results
The first 600 items, on the Aillio Bullet R1 and R2, whose simulators were ready first:
| Model | Setup | Median | Mean |
|---|---|---|---|
| GPT-6 Luna | harness | 100.0 | 99.2 |
| Aden Experimental | harness | 98.3 | 88.0 |
| Claude Haiku 5.5 | harness | 97.6 | 96.4 |
| DeepSeek V4.1 Flash | single pass | 97.6 | 78.4 |
| Aden Experimental | single pass | 97.6 | 82.7 |
| Claude Opus 5.5 | single pass | 89.5 | 82.8 |
| GPT-6 Luna | single pass | 86.1 | 65.9 |
| Claude Haiku 5.5 | single pass | 81.8 | 63.7 |
| Qwen3.5-9B, Aden Experimental's base | single pass | 19.8 | 34.7 |
| The human reference program | 97.9 | 92.8 |
- Single pass is one answer, with no help. With the harness, a model may compile, roast and grade up to six drafts before its one graded answer, so the two setups aren't comparable.
- The medians hide the zeros. A roast that never cracks scores nothing, and the means show how often that happens. Most misses stall short of crack, with the heat cut low and the fan high.
- The human reference is each item's original roast, run on the same simulator and graded the same way.
The other roasters join as their simulators pass their fits.