Introduction to BenchCAD — BenchCAD Blog

what it is

A benchmark for multimodal models

BenchCAD is a benchmark for multimodal models (MLLMs). The task sounds simple — look at a mechanical part and write the CAD code that rebuilds it — but doing it well takes the full multimodal stack: visual perception, image–text–code alignment, and 3D spatial reasoning.

A model has to read the views, tie them to text and code, reason about the shape in 3D, and recover the exact numbers. Two programs can look identical yet differ in dimensions, holes, or symmetry — so every answer is checked against real geometry, not appearances. The geometry passes the eye but fails the caliper.

Four properties make it a clean test of multimodal spatial reasoning:

Execution-verified at scale — 17,900 sandbox-executed CadQuery parts across 106 industrial families.
Grounded in real parts — 106 industrial families; about half (52/106) follow ISO / DIN / EN / ASME / IEC standards, the rest typical engineering practice.
Spatially rich — 46 CadQuery operations (rotations, arrays, sweeps, lofts, helices) that need genuine 3D reasoning.
Capability-decomposed — four matched tasks (Vision2Code · Vision QA · Code QA · Code Edit) separating perception, image↔code grounding, numeric reasoning, and code synthesis.

data

Three Released Datasets

Every family hand-crafted by domain experts from industrial standards.

BenchCAD 17,900

Verified CadQuery parts — code · STEP · 4 canonical views · parameters · operation traces.

BenchCAD-QA 2,400

Paired image / code numeric QA items along a four-level capability hierarchy.

BenchCAD-Edit 748

Verified before / after edit pairs across five edit types T1–T5.

BenchCAD family distribution — 106 part families across industrial sectors

106 industrial part families — fasteners, transmission, structural, fluid, panels, hardware, enclosures. About half (52/106) sample from real specification tables across 47 ISO / DIN / EN / ASME / IEC codes; the rest are common engineering parts.

evaluation

Capability Hierarchy

The same questions are evaluated under Vision QA (renders) and Code QA (source); the matched pair exposes a modality gap — isolating whether a failure stems from visual perception or from parametric reasoning. CAD errors are often compositional: right family but wrong operation, right scale but wrong standard relation.

L1 Holistic Visual Recognition · L2 CAD Operation Understanding · L3 Industrial Parametric Abstraction · L4 Compositional Spatial / Code Reasoning. Scoring is execution-grounded — IoU for geometry, ratio accuracy for QA.

cite

BibTeX

bibtex

@misc{benchcad2026,
  title        = {BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD},
  author       = {Zhang, Haozhe and Liu, Kaichen and Chen, Miaomiao and Li, Lei
                  and Yang, Shaojie and Peng, Cheng and Chen, Hanjie},
  year         = {2026},
  eprint       = {2605.10865},
  archivePrefix= {arXiv},
  primaryClass = {cs.CV},
  url          = {https://arxiv.org/abs/2605.10865}
}