# Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models > Cybench turns 40 professional-level capture-the-flag tasks from four competitions (HackTheBox Cyber Apocalypse 2024, SekaiCTF 2022–23, GlacierCTF and HKCert) into runnable agent tasks. Each task has a description, starter files, a Kali Linux container for the agent with task servers in Docker, and the time the first human team needed to solve it, from 2 minutes to 24 hours 54 minutes. Each task is also split into subtasks that grade intermediate steps. The authors evaluate eight models with a structured bash agent and compare four scaffolds for GPT-4o and Claude 3.5 Sonnet. Without subtask guidance, the best agents solved only tasks whose first human solve took at most 11 minutes, and Claude 3.5 Sonnet reached 17.5% unguided success. - Authors: Andy K. Zhang; Neil Perry; Riya Dulepet; Joey Ji; Celeste Menders; Justin W. Lin; Eliot Jones; Gashon Hussein; Samantha Liu; Donovan Jasper; Pura Peetathawatchai; Ari Glenn; Vikram Sivashankar; Daniel Zamoshchin; Leo Glikbarg; Derek Askaryar; Mike Yang; Teddy Zhang; Rishi Alluri; Nathan Tran; Rinnara Sangpisit; Polycarpos Yiorkadjis; Kenny Osele; Gautham Raghupathi; Dan Boneh; Daniel E. Ho; Percy Liang - Venue: ICLR 2025 - Status: Conference paper - Publication / posting date: 2025 (exact date not verified) - Date source: Publication year stated by the official ICLR proceedings; no finer publication date is asserted. The proceedings page's citation date of 2025-05-01 is not adopted, and the arXiv dates are not substituted. - First arXiv submission: August 15, 2024 - arXiv: 2408.08926 - Reviewed arXiv version: v4 - Sources reviewed: 2026-10-19 - Added to the catalog: 2026-10-27 - Original paper: - PDF: - Code: - Project: - BibTeX: - Catalog page: - Record JSON: - Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-cybench/index.md) ## Scope and limitations Only 6 of the 40 tasks are reverse engineering; the rest are cryptography (16), web (8), forensics (4), miscellaneous (4) and pwn (2), so aggregate rates are not reverse-engineering accuracy, and Cybench is not a reverse-engineering-only benchmark. In the unguided structured-bash runs, six or seven of the eight models solved the two easiest reversing tasks (first human solves in 2 and 6 minutes), and none solved the other four. Results cover 2024 models (GPT-4o, o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 and 3.1, Mixtral 8x22b) with the authors' scaffolds: 15 iterations unguided or 5 per subtask, short memory, no decompiler tools, one attempt in the main comparison and the best of three in the scaffold comparison. First-solve times come from competition records and are not directly comparable across competitions. The authors judge train–test overlap by model knowledge cutoffs and cannot rule it out for Claude 3.5 Sonnet. CTF tasks are short, small-codebase competition problems, and capturing a flag does not show recovered program structure. ## Why it fits this collection Adds a CTF agent benchmark with reverse-engineering tasks and subtask-level grading; EnIGMA in this section reports results on it, and it complements the reverse-engineering-specific benchmarks here. ## Publication and review notes Authors follow the byline of the proceedings PDF, which matches arXiv v4. The proceedings page metadata instead lists 25 names: it omits Polycarpos Yiorkadjis and Gautham Raghupathi, gives Haoxiang Yang, Aolin Zhang and Kenny Oseleononmen where the PDF prints Mike Yang, Teddy Zhang and Kenny Osele, and drops middle initials. The proceedings list pages 25094–25243 and no DOI. arXiv v1 is dated 2024-08-15, v2 2024-10-06, v3 2024-12-05 and v4 2025-04-12; v4 carries the comment 'ICLR 2025 Oral'. The OpenReview forum could not be retrieved past a browser check, so the oral designation rests on that comment. The proceedings abstract links anonymized Google Drive archives, while the PDF points to cybench.github.io, which links andyzorigin/cybench. At the reviewed 2026-09-24 commit the repository is Apache-2.0; runs need model API keys, Python 3.9–3.10 and Docker with the cybench/cybench image. Its task_list.txt names all 40 tasks, but the nine GlacierCTF tasks, including the reversing tasks RPGO and SOP, are not in the repository tree. ## Primary sources 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. ## About this entry Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX. Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.