projects · live
BenchCAD 1.0
The first version of BenchCAD: single manufacturable parts reconstructed from multi-view renders, graded by executing the program and comparing geometry. IoU-score = voxel IoU × exec%.
- 17,900 execution-verified CadQuery parts across 106 industrial families — about half anchored to real ISO / DIN / EN / ASME / IEC specification tables (47 codes).
- Four matched tasks on the same parts: Vision2Code (the flagship), Vision QA, Code QA, Code Edit.
- Deterministic grading — geometry by voxel IoU, QA by ratio accuracy. No LLM judge.
- Reported by labs: Anthropic has reported Vision2Code in its system cards since June 2026, and OpenAI in its GPT-5.6 and GPT-6 Astra launches. With tools, the best reported score is now 0.963 (Claude Sonnet 5.5). See the leaderboard and news.
- Paper: arXiv:2605.10865.
Contribute a part family
BenchCAD grows by family — a parametric CadQuery generator for a real industrial part, shipped with QA and edit pairs:
- Start from CONTRIBUTING.md and the example in
contributions/<family>/(family.json·generator.py·qa/·edits/). - Validate locally with
tools/validate_task.pyandtools/validate_family.py— no API keys needed. - Maintainers re-check and ingest accepted families into the HuggingFace dataset — every accepted family is credited on the contributors wall.
Running a model instead? See submit a model to the leaderboard.