BenchCAD 2.0

Professional, complex hardware engineering work: 3D parts and assemblies built from engineering drawings or rendered images, and circuit wiring reconstructed from renders of physical boards. Each task is evaluated separately against references, without an LLM judge.

Drawing to CAD: a part from a dimensioned part drawing; an assembly from every part as a 3D file plus the assembly drawing; parts and an assembly from drawings of the parts to make and 3D files of the parts to buy.
Reverse engineering: a part from four rendered views; parts and an assembly from views of the assembly and of each part; a wiring graph (ECAD) from six renders of an assembled board.
Core (held-out): 100 cases across all six tasks, behind every score we report. Researchers and model developers can evaluate on it under an evaluation-only agreement.
Public: 100 open cases across all six tasks, with inputs and references, none of them in Core. Anyone can download them to try the harness and see the tasks, after accepting evaluation-only terms on Hugging Face (instant, no review); scores on them are not reportable.
Harness: up to 30 rounds per case in an offline sandbox with Python's standard library, CadQuery, NumPy, SciPy and an image-cropping tool, but no built-in rendering tool; models can write their own.
Results: five frontier models on Core at every reasoning-effort setting. The best model scores 0.59 out of 1. See the charts and the board on the home page.

Contributors