BenchCAD 2.1 Core

Core, 100 held-out cases across six tasks, at every reasoning effort. Two boards: the standard 2.1 harness, and customized harnesses such as a lab's own agent. Scores run from 0 to 1.

To add a line, run Core with the v2.1 tag and send us core_summary.json; see Core access and Setup. More in the 2.1 post.

BenchCAD 2.0 Core

Five frontier models on Core, 100 held-out cases across six tasks, at every reasoning effort, each with up to 30 rounds per case. One row per model and effort; scores run from 0 to 1.

2026-11-05: in harness v2.0, one GPT-6 Astra (max) Task 6 case was scored 0.50 from a graph left in the working directory, although the model never submitted. Under the prompt's rule a case with no submit scores 0, which would put Astra (max) at 0.589 instead of 0.594 (0.59 in this table either way). No other line changes at this table's precision. Published numbers are left as run; harness v2.1 fixes this.

To add a model, run Core at the published settings and send us core_summary.json; see Core access and Setup. More in the 2.0 post.

Vision2Code

* Self-reported — voxel IoU only, not re-graded. Anthropic: Mythos 5 / Mythos Preview / Opus 4.8 on the full set (Fable 5 / Mythos 5 system card), Sonnet 4.6 on a random 1,000-file subset (Sonnet 5 system card, fig. 8.10.3.A), Haiku 5.5 and Haiku 4.5 on a random 1,000-file subset (Haiku 5.5 system card, fig. 8.9.2.A; Haiku 4.5 at fixed extended-thinking budgets up to 32k); Sonnet 5.5, Opus 5.5, Fable 5.1, Opus 5 and Sonnet 5 on a random 1,000-file subset (Sonnet 5.5 system card and Opus 5.5 system card, fig. 8.13.2.A; Fable 5.1 first reported in the Fable 5.1 / Mythos 5.1 system card, fig. 8.14.2.A). OpenAI: GPT-5.5 and GPT-5.6 Sol / Terra / Luna, no tools (GPT-5.6 launch table). Moonshot: Kimi K3.1 Pro on a random 1,000-file subset, max effort, single-shot (K3.1 Pro launch post).

IoU · tools — the agentic setting: the model gets a Python sandbox to render, measure and iterate before submitting. Anthropic and OpenAI figures are vendor self-reported voxel IoU, not re-graded. Anthropic — a random 1,000-file subset, not the same split as the IoU-score beside it: Haiku 5.5 0.870 / Haiku 4.5 0.210 (Haiku 5.5 system card, fig. 8.9.2.A); Sonnet 5.5 0.963 (Sonnet 5.5 system card, fig. 8.13.2.A) / Opus 5.5 0.962 / Fable 5.1 0.926 / Opus 5 0.899 / Sonnet 5 0.519 (Opus 5.5 system card, fig. 8.13.2.A; Fable 5.1 first reported in the Fable 5.1 / Mythos 5.1 system card, fig. 8.14.2.A), max effort, adaptive thinking, averaged over five runs; Mythos 5 0.678 / Opus 4.8 0.521 (Opus 5 system card, fig. 8.12.2.A); Mythos Preview 0.611 / Sonnet 4.6 0.327 (Sonnet 5 system card, fig. 8.10.3.A). Each is the latest score Anthropic has published for that model. OpenAI — GPT-5.5 and GPT-5.6 Sol / Terra / Luna with a Python tool (GPT-5.6 launch table); split and attempt budget unstated. Moonshot — Kimi K3.1 Pro 0.927, max effort in the Claude Code harness, self-reported on a random 1,000-file subset, the same split as its IoU-score (K3.1 Pro launch post). SpaceXAI — Grok 4.6 0.8055 (xhigh) and Grok 4.5 0.7771 (high), our own agentic runs on our scorer, not vendor-reported. GPT-6 Astra 0.959 — reported in OpenAI's GPT-6 Astra launch coverage (launch post); the same table quotes Fable 5.1 0.843, Opus 5 0.821 and Fable 5 0.675, Anthropic's figures at the time. Secondhand: OpenAI's page was unreachable when this row was added and no no-tools figure has been published.

To add a model to the this leaderboard, run the command above and open a Model result submission issue on the code repo with your submission.jsonl. Submissions are re-graded before listing.