About BenchCAD
BenchCAD is a benchmark for multimodal models (MLLMs). The task sounds simple — look at a mechanical part and write the CAD code that rebuilds it — but doing it well takes the full multimodal stack: visual perception, image–text–code alignment, and 3D spatial reasoning.
A model has to read the views, tie them to text and code, reason about the shape in 3D, and recover the exact numbers. Two programs can look identical yet differ in dimensions, holes, or symmetry — so every answer is checked against real geometry, not appearances. The geometry passes the eye but fails the caliper.
Four properties make it a clean test of multimodal spatial reasoning:
- Execution-verified at scale — 17,900 sandbox-executed CadQuery parts across 106 industrial families.
- Grounded in real parts — 106 industrial families; about half (52/106) follow ISO / DIN / EN / ASME / IEC standards, the rest typical engineering practice.
- Spatially rich — 46 CadQuery operations (rotations, arrays, sweeps, lofts, helices) that need genuine 3D reasoning.
- Capability-decomposed — four matched tasks (Vision2Code · Vision QA · Code QA · Code Edit) separating perception, image↔code grounding, numeric reasoning, and code synthesis.
Three Released Datasets
Every family hand-crafted by domain experts from industrial standards.
17,900Verified CadQuery parts — code · STEP · 4 canonical views · parameters · operation traces.
2,400Paired image / code numeric QA items along a four-level capability hierarchy.
748Verified before / after edit pairs across five edit types T1–T5.
106 industrial part families — fasteners, transmission, structural, fluid, panels, hardware, enclosures. About half (52/106) sample from real specification tables across 47 ISO / DIN / EN / ASME / IEC codes; the rest are common engineering parts.
Capability Hierarchy
The same questions are evaluated under Vision QA (renders) and Code QA (source); the matched pair exposes a modality gap — isolating whether a failure stems from visual perception or from parametric reasoning. CAD errors are often compositional: right family but wrong operation, right scale but wrong standard relation.
L1 Holistic Visual Recognition · L2 CAD Operation Understanding · L3 Industrial Parametric Abstraction · L4 Compositional Spatial / Code Reasoning. Scoring is execution-grounded — IoU for geometry, ratio accuracy for QA.
BibTeX
@misc{benchcad2026,
title = {BenchCAD: A Comprehensive, Industry-Standard Benchmark for Programmatic CAD},
author = {Zhang, Haozhe and Liu, Kaichen and Chen, Miaomiao and Li, Lei
and Yang, Shaojie and Peng, Cheng and Chen, Hanjie},
year = {2026},
eprint = {2605.10865},
archivePrefix= {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2605.10865}
}Dataset ships with Croissant 1.0 metadata · code MIT · data CC-BY-4.0.