Setup
Run BenchCAD 2.0 or BenchCAD 1.0 on your own model.
BenchCAD 2.0
One script installs the environment, builds the sandbox, downloads and verifies the dataset, runs every case and scores it. It needs Docker and a Hugging Face token; Core also needs Core access. The full options are in the code repo.
git clone https://github.com/BenchCAD-org/benchcad-2-main && cd benchcad-2-main
export HF_TOKEN=... ANTHROPIC_API_KEY=... # or OPENAI_API_KEY / GEMINI_API_KEY
# the open Public set: try the harness and look at cases (scores are not reportable)
tools/run_core.sh --model <provider/model> --set public --effort low
# Core: the 100 cases behind the reported scores
tools/run_core.sh --model <provider/model> --set coreBenchCAD 1.0
Install
BenchCAD runs on Python through uv.
Clone the repo, sync the environment, then add the API key for the model you want to evaluate.
git clone https://github.com/BenchCAD/BenchCAD-main
cd BenchCAD-main
uv sync # create the environment
cp .env.example .env # paste OpenAI / Anthropic / Gemini / OpenRouter keysRun the benchmark
One command runs all four BenchCAD tasks — Vision2Code, Vision QA, Code QA,
and Code Edit — pulling data from HuggingFace on demand. --model is required;
a bare run is a 5-record-per-task smoke, and --num all runs the full split.
uv run python benchcad.py --model gpt-4o # smoke — all tasks, 5 records each
uv run python benchcad.py --task all --num all --model gpt-4o # full benchmark
uv run python benchcad.py --task vision2code --num 100 --model claude-opus-4-7 --seed 42
--task takes all · vision2code · codeedit · qa
(qa covers both Vision QA and Code QA), and --model accepts several names at
once. Each task also runs standalone: cd Vision2Code && uv run python main.py --config configs/prod.yaml --model gpt-4o.