blog · 2026-11-05

BenchCAD 2.1

A new standard harness for Core, and a track for customized harnesses.

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 $0.01 $0.03 $0.1 $0.3 $1 $3 $10 $30 $100 cost per case, USD (log scale) Fable 5.1 · credit to Anthropic · low effort · score 0.082 ± 0.008 at $1.49 per case Fable 5.1 · credit to Anthropic · medium effort · score 0.121 ± 0.007 at $3.18 per case Fable 5.1 · credit to Anthropic · high effort · score 0.204 ± 0.010 at $6.94 per case Fable 5.1 · credit to Anthropic · xhigh effort · score 0.340 ± 0.022 at $16.52 per case Fable 5.1 · credit to Anthropic · max effort · score 0.363 ± 0.022 at $19.98 per case Haiku 5.5 · credit to Anthropic · low effort · score 0.049 ± 0.004 at $0.0078 per case Haiku 5.5 · credit to Anthropic · medium effort · score 0.060 ± 0.003 at $0.014 per case Haiku 5.5 · credit to Anthropic · high effort · score 0.091 ± 0.007 at $0.061 per case Haiku 5.5 · credit to Anthropic · xhigh effort · score 0.149 ± 0.007 at $0.47 per case Haiku 5.5 · credit to Anthropic · max effort · score 0.186 ± 0.016 at $1.35 per case Sonnet 5.5 · credit to Anthropic · low effort · score 0.052 ± 0.005 at $0.076 per case Sonnet 5.5 · credit to Anthropic · medium effort · score 0.061 ± 0.006 at $0.11 per case Sonnet 5.5 · credit to Anthropic · high effort · score 0.161 ± 0.007 at $1.04 per case Sonnet 5.5 · credit to Anthropic · xhigh effort · score 0.370 ± 0.017 at $3.46 per case Sonnet 5.5 · credit to Anthropic · max effort · score 0.468 ± 0.014 at $6.07 per case Opus 5.5 · credit to Anthropic · low effort · score 0.069 ± 0.006 at $0.15 per case Opus 5.5 · credit to Anthropic · medium effort · score 0.151 ± 0.013 at $0.89 per case Opus 5.5 · credit to Anthropic · high effort · score 0.243 ± 0.013 at $2.06 per case Opus 5.5 · credit to Anthropic · xhigh effort · score 0.424 ± 0.022 at $5.23 per case Opus 5.5 · credit to Anthropic · max effort · score 0.490 ± 0.013 at $8.10 per case Fable 5.2 · Anthropic's run of the 2.1 protocol, from the Fable 5.2 system card · credit to Anthropic · low effort · score ≈0.213 ≈$1.3 per case (estimated from the system card's chart) Fable 5.2 · Anthropic's run of the 2.1 protocol, from the Fable 5.2 system card · credit to Anthropic · medium effort · score ≈0.286 ≈$2.2 per case (estimated from the system card's chart) Fable 5.2 · Anthropic's run of the 2.1 protocol, from the Fable 5.2 system card · credit to Anthropic · high effort · score ≈0.592 ≈$9.4 per case (estimated from the system card's chart) Fable 5.2 · Anthropic's run of the 2.1 protocol, from the Fable 5.2 system card · credit to Anthropic · xhigh effort · score ≈0.702 ≈$17.1 per case (estimated from the system card's chart) Fable 5.2 · Anthropic's run of the 2.1 protocol, from the Fable 5.2 system card · credit to Anthropic · max effort · score 0.754 ≈$23.3 per case (estimated from the system card's chart) Fable 5.1 Haiku 5.5 Sonnet 5.5 Opus 5.5 Fable 5.2 score

Figure 1. Score on Core against cost per case, with the 2.1 harness. Each point is one model at one reasoning effort, the mean of 5 passes over all 100 Core cases. Credit to Anthropic. Claude Fable 5.2: from Anthropic's Fable 5.2 system card (§8.13.2, Anthropic's implementation of the 2.1 protocol): max effort from the card's text, lower efforts ≈ read off its chart; no confidence interval given, cost estimated from the chart.

TL;DR
  • BenchCAD 2.1 lets models use tokens more efficiently, and adds a separate board for customized harnesses.
  • Claude Fable 5.2, from Anthropic's system card, scores 0.754 at max on the standard 2.1 harness and 0.835 in Claude Code, the highest Claude scores so far.
  • In Anthropic's 5-pass run of the other four Claude models, Opus 5.5 leads from medium effort up: 0.49 at max for $8.1 per case on the standard harness, and 0.65 in Claude Code.
  • OpenAI's GPT-6.2 Astra, run in Codex, scores 0.825 at max for $5.43 per case, and GPT-6.2 Sol 0.731 for $1.60 (credit to OpenAI).

What changed

2.02.1
Rounds per case3060
Output tokens per caseno budget500,000, thinking included
Reasoning across roundsdroppedcarried
The prompt states the limitsnoyes, with rounds and tokens spent so far
The prompt says no submit scores 0noyes
A discarded, re-asked reply counts against the budget—no

These limits are for the standard 2.1 harness. The customized-harness track (Claude Code, Codex) has no token or time limit.

Scores from 2.1 are not comparable with 2.0.

Task 6 also matches components more strictly: a prediction can only claim the part nearest to it, and two components answered at the same spot both count as wrong (how Task 6 is scored).

The case files are unchanged, and 2.0 still reproduces from its own tag; details are in the release notes.

Results

Credit to Anthropic for the Claude results. With the 2.1 harness, 5 passes per model, Claude Opus 5.5 leads the four models in that run from medium effort up, with 0.49 at max. In Claude Code, Opus 5.5 reaches 0.65. Both boards are on the leaderboard.

Added 2026-11-09: Claude Fable 5.2, from Anthropic's system card (§8.13.2). At max effort it scores 0.754 with Anthropic's own run of the 2.1 protocol and 0.835 with tools. The lower efforts in the charts are read off the card's chart (≈).

Added 2026-11-17: OpenAI's post Introducing GPT-6.2 Astra and Sol reports BenchCAD 2.1 Core at every effort. At max, GPT-6.2 Astra scores 0.825 for $5.43 per case, GPT-6.2 Sol 0.731 for $1.60, GPT-6 Astra 0.626 and GPT-6.1 Sol 0.526. Run in Codex, credit to OpenAI; they're on the customized-harness board (Figure 1, second tab).

Acknowledgments

We thank Anthropic for the Claude results, and OpenAI for the GPT results.

Citation

2.1 is a new release of the same benchmark, so please cite BenchCAD 2.0, and say in your paper that you used release 2.1 (tag v2.1):

@misc{benchcad2_2026,
  title        = {BenchCAD 2.0: Benchmarking Programmatic CAD on Real Engineering Work},
  author       = {{BenchCAD Team}},
  year         = {2026},
  howpublished = {\url{https://benchcad.com/blog/benchcad-2.html}},
  note         = {Blog post}
}