Setup
Install BenchCAD and run all four tasks locally. Every number on the leaderboard is produced this way — by re-executing model output, never self-reported.
Install
BenchCAD runs on Python through uv.
Clone the repo, sync the environment, then add the API key for the model you want to evaluate.
git clone https://github.com/BenchCAD/BenchCAD-main
cd BenchCAD-main
uv sync # create the environment
cp .env.example .env # paste OpenAI / Anthropic / Gemini / OpenRouter keysRun the benchmark
One command runs all four BenchCAD tasks — Vision2Code, Vision QA, Code QA,
and Code Edit — pulling data from HuggingFace on demand. --model is required;
a bare run is a 5-record-per-task smoke, and --num all runs the full split.
uv run python benchcad.py --model gpt-4o # smoke — all tasks, 5 records each
uv run python benchcad.py --task all --num all --model gpt-4o # full benchmark
uv run python benchcad.py --task vision2code --num 100 --model claude-opus-4-7 --seed 42
--task takes all · vision2code · codeedit · qa
(qa covers both Vision QA and Code QA), and --model accepts several names at
once. Each task also runs standalone: cd Vision2Code && uv run python main.py --config configs/prod.yaml --model gpt-4o.