What is CWE-bench v2 and how does it work?

CWE-bench v2 is a comprehensive benchmarking framework designed to test the limits of frontier cyber capabilities in modern language models. Developed by Collinear AI, it provides a public set of 100 tasks that anyone can run and evaluate to measure how effectively models can find and fix security weaknesses in real open-source repositories.

The scoring mechanism is fully programmatic, ensuring deterministic results that can be reproduced without extra cost. All checks have been curated using LLM judge panels and reviewed by hand to guarantee high quality, realism, and fairness.

Which models and tasks are affected by the benchmark?

These 100 tasks cover 91 different Common Weakness Enumeration classes, spanning a wide range of vulnerability surfaces. A model attempting these tasks must truly understand multiple exploitation vectors rather than just recognizing a familiar pattern.

Evaluations show that Gemini 4 Argon leads the pack with 67.8% pass@1 and 81.0% pass@4 amid intense hardware demands, closely followed by Claude Opus 5.5 at 67.3% pass@1. Despite these strong scores, even the top model fails 19% of the set on every attempt, proving that the long tail of security challenges remains far from solved even with advanced optimizations.

How can I install and run CWE-bench v2?

Every task in CWE-bench v2 is self-contained as a Harbor task, consisting of a Dockerized environment, instructions, and a deterministic verifier. You can run the entire benchmark straight from the Hugging Face repository using Harbor.

Make sure to accept the dataset access conditions on the Hugging Face page first, then execute the following steps to run your evaluation:

  1. Save your Hugging Face token to git's credential helper by running: hf auth login --token $HF_TOKEN --add-to-git-credential
  2. Execute the Harbor command pointing to the repository: harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4 --agent <agent> --model <model> --env docker --allow-agent-host <model API host>
  3. Review the reported scores, which calculate the mean reward over all 400 trials (-k 4 runs 4 attempts per task).