BenchCAD
blog · 2026-11-05

BenchCAD 2.1

A new standard harness for Core, and a track for customized harnesses.

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 $0.01 $0.03 $0.1 $0.3 $1 $3 $10 $30 cost per case, USD (log scale) Fable 5.1 · credit to Anthropic · low effort · score 0.082 ± 0.008 at $1.49 per case Fable 5.1 · credit to Anthropic · medium effort · score 0.121 ± 0.007 at $3.18 per case Fable 5.1 · credit to Anthropic · high effort · score 0.204 ± 0.010 at $6.94 per case Fable 5.1 · credit to Anthropic · xhigh effort · score 0.340 ± 0.022 at $16.52 per case Fable 5.1 · credit to Anthropic · max effort · score 0.363 ± 0.022 at $19.98 per case Haiku 5.5 · credit to Anthropic · low effort · score 0.049 ± 0.004 at $0.0078 per case Haiku 5.5 · credit to Anthropic · medium effort · score 0.060 ± 0.003 at $0.014 per case Haiku 5.5 · credit to Anthropic · high effort · score 0.091 ± 0.007 at $0.061 per case Haiku 5.5 · credit to Anthropic · xhigh effort · score 0.149 ± 0.007 at $0.47 per case Haiku 5.5 · credit to Anthropic · max effort · score 0.186 ± 0.016 at $1.35 per case Sonnet 5.5 · credit to Anthropic · low effort · score 0.052 ± 0.005 at $0.076 per case Sonnet 5.5 · credit to Anthropic · medium effort · score 0.061 ± 0.006 at $0.11 per case Sonnet 5.5 · credit to Anthropic · high effort · score 0.161 ± 0.007 at $1.04 per case Sonnet 5.5 · credit to Anthropic · xhigh effort · score 0.370 ± 0.017 at $3.46 per case Sonnet 5.5 · credit to Anthropic · max effort · score 0.468 ± 0.014 at $6.07 per case Opus 5.5 · credit to Anthropic · low effort · score 0.069 ± 0.006 at $0.15 per case Opus 5.5 · credit to Anthropic · medium effort · score 0.151 ± 0.013 at $0.89 per case Opus 5.5 · credit to Anthropic · high effort · score 0.243 ± 0.013 at $2.06 per case Opus 5.5 · credit to Anthropic · xhigh effort · score 0.424 ± 0.022 at $5.23 per case Opus 5.5 · credit to Anthropic · max effort · score 0.490 ± 0.013 at $8.10 per case Fable 5.1 Haiku 5.5 Sonnet 5.5 Opus 5.5 score

Figure 1. Score on Core against cost per case, with the 2.1 harness. Each point is one model at one reasoning effort, the mean of 5 passes over all 100 Core cases. Credit to Anthropic.

TL;DR
  • BenchCAD 2.1 lets models use tokens more efficiently, and adds a separate board for customized harnesses.
  • On the standard 2.1 harness, Claude Opus 5.5 leads at every reasoning effort from medium up, and scores 0.49 at max for $8.1 per case.
  • In Claude Code, Opus 5.5 reaches 0.65 at max, the highest of the four Claude models.

What changed

2.02.1
Rounds per case3060
Output tokens per caseno budget500,000, thinking included
Reasoning across roundsdroppedcarried
The prompt states the limitsnoyes, with rounds and tokens spent so far
The prompt says no submit scores 0noyes
A discarded, re-asked reply counts against the budget—no

Scores from 2.1 are not comparable with 2.0.

Task 6 also matches components more strictly: a prediction can only claim the part nearest to it, and two components answered at the same spot both count as wrong (how Task 6 is scored).

The case files are unchanged, and 2.0 still reproduces from its own tag; details are in the release notes.

Results

Credit to Anthropic for the Claude results. With the 2.1 harness, 5 passes per model, Claude Opus 5.5 leads from medium effort up, with 0.49 at max. In Claude Code, Opus 5.5 reaches 0.65. Both boards are on the leaderboard.

GPT-6 Astra on 2.1: coming soon.

Acknowledgments

We thank Anthropic for the Claude results.

Citation

2.1 is a new release of the same benchmark, so please cite BenchCAD 2.0, and say in your paper that you used release 2.1 (tag v2.1):

@misc{benchcad2_2026,
  title        = {BenchCAD 2.0: Benchmarking Programmatic CAD on Real Engineering Work},
  author       = {{BenchCAD Team}},
  year         = {2026},
  howpublished = {\url{https://benchcad.com/blog/benchcad-2.html}},
  note         = {Blog post}
}