{ "added_at": "2026-10-27", "arxiv_id": "2408.08926", "arxiv_version_reviewed": "v4", "authors": [ "Andy K. Zhang", "Neil Perry", "Riya Dulepet", "Joey Ji", "Celeste Menders", "Justin W. Lin", "Eliot Jones", "Gashon Hussein", "Samantha Liu", "Donovan Jasper", "Pura Peetathawatchai", "Ari Glenn", "Vikram Sivashankar", "Daniel Zamoshchin", "Leo Glikbarg", "Derek Askaryar", "Mike Yang", "Teddy Zhang", "Rishi Alluri", "Nathan Tran", "Rinnara Sangpisit", "Polycarpos Yiorkadjis", "Kenny Osele", "Gautham Raghupathi", "Dan Boneh", "Daniel E. Ho", "Percy Liang" ], "bibtex_url": "https://awesomerea.org/papers/citations/paper-cybench.bib", "code_url": "https://github.com/andyzorigin/cybench", "date_basis": "Publication year stated by the official ICLR proceedings; no finer publication date is asserted. The proceedings page's citation date of 2025-05-01 is not adopted, and the arXiv dates are not substituted.", "date_basis_zh": "采用 ICLR 官方论文集标明的出版年份，不主张更精确的出版日期。本条目不采用论文集页面中 2025-05-01 的引用日期，也不以 arXiv 日期替代。", "date_precision": "year", "description_en": "Cybench turns 40 professional-level capture-the-flag tasks from four competitions (HackTheBox Cyber Apocalypse 2024, SekaiCTF 2022–23, GlacierCTF and HKCert) into runnable agent tasks. Each task has a description, starter files, a Kali Linux container for the agent with task servers in Docker, and the time the first human team needed to solve it, from 2 minutes to 24 hours 54 minutes. Each task is also split into subtasks that grade intermediate steps. The authors evaluate eight models with a structured bash agent and compare four scaffolds for GPT-4o and Claude 3.5 Sonnet. Without subtask guidance, the best agents solved only tasks whose first human solve took at most 11 minutes, and Claude 3.5 Sonnet reached 17.5% unguided success.", "description_zh": "Cybench 将来自四项竞赛（HackTheBox Cyber Apocalypse 2024、SekaiCTF 2022–23、GlacierCTF 与 HKCert）的 40 道专业级夺旗赛题目整理为可运行的智能体任务。每道题目都包含题目描述、初始文件、供智能体使用的 Kali Linux 容器及以 Docker 运行的题目服务器，并记录首个人类队伍的解题用时（2 分钟至 24 小时 54 分钟）。每道题目还被拆分为若干子任务，用于评估中间步骤。作者使用结构化 bash 智能体评测了八个模型，并为 GPT-4o 与 Claude 3.5 Sonnet 比较了四种智能体框架。在没有子任务引导时，表现最好的智能体只解出了人类首解用时不超过 11 分钟的题目；Claude 3.5 Sonnet 的无引导成功率为 17.5%。", "doi": null, "entry_url": "https://awesomerea.org/papers/paper-cybench/", "entry_url_zh": "https://awesomerea.org/zh/papers/paper-cybench/", "evidence_urls": [ "https://proceedings.iclr.cc/paper_files/paper/2025/hash/3e9412a9c1d93810ef3ef7825115016b-Abstract-Conference.html", "https://proceedings.iclr.cc/paper_files/paper/2025/file/3e9412a9c1d93810ef3ef7825115016b-Paper-Conference.pdf", "https://arxiv.org/abs/2408.08926", "https://arxiv.org/pdf/2408.08926v4", "https://cybench.github.io/", "https://github.com/andyzorigin/cybench/blob/ca483c6443f17a20ba7d6bac3d274859a1721e51/README.md", "https://github.com/andyzorigin/cybench/blob/ca483c6443f17a20ba7d6bac3d274859a1721e51/LICENSE", "https://github.com/andyzorigin/cybench/blob/ca483c6443f17a20ba7d6bac3d274859a1721e51/task_list.txt", "https://github.com/andyzorigin/cybench/tree/ca483c6443f17a20ba7d6bac3d274859a1721e51/benchmark", "https://github.com/andyzorigin/cybench/blob/ca483c6443f17a20ba7d6bac3d274859a1721e51/run_task.sh" ], "first_arxiv_date": "2024-08-15", "id": "paper-cybench", "json_url": "https://awesomerea.org/papers/paper-cybench/record.json", "kind": "research", "markdown_url": "https://awesomerea.org/papers/paper-cybench/index.md", "markdown_url_zh": "https://awesomerea.org/zh/papers/paper-cybench/index.md", "metadata_notes": "Authors follow the byline of the proceedings PDF, which matches arXiv v4. The proceedings page metadata instead lists 25 names: it omits Polycarpos Yiorkadjis and Gautham Raghupathi, gives Haoxiang Yang, Aolin Zhang and Kenny Oseleononmen where the PDF prints Mike Yang, Teddy Zhang and Kenny Osele, and drops middle initials. The proceedings list pages 25094–25243 and no DOI. arXiv v1 is dated 2024-08-15, v2 2024-10-06, v3 2024-12-05 and v4 2025-04-12; v4 carries the comment 'ICLR 2025 Oral'. The OpenReview forum could not be retrieved past a browser check, so the oral designation rests on that comment. The proceedings abstract links anonymized Google Drive archives, while the PDF points to cybench.github.io, which links andyzorigin/cybench. At the reviewed 2026-09-24 commit the repository is Apache-2.0; runs need model API keys, Python 3.9–3.10 and Docker with the cybench/cybench image. Its task_list.txt names all 40 tasks, but the nine GlacierCTF tasks, including the reversing tasks RPGO and SOP, are not in the repository tree.", "metadata_notes_zh": "作者以论文集 PDF 的署名为准，与 arXiv v4 一致。论文集页面元数据则列出 25 人：缺少 Polycarpos Yiorkadjis 与 Gautham Raghupathi，在 PDF 印作 Mike Yang、Teddy Zhang 与 Kenny Osele 之处写作 Haoxiang Yang、Aolin Zhang 与 Kenny Oseleononmen，并省略了中间名缩写。论文集列出页码 25094–25243，没有 DOI。arXiv v1 发布于 2024-08-15，v2 为 2024-10-06，v3 为 2024-12-05，v4 为 2025-04-12；v4 注释为“ICLR 2025 Oral”。OpenReview 论坛页因浏览器验证无法获取，因此口头报告身份依据该注释。论文集摘要链接的是匿名 Google Drive 压缩包，而 PDF 指向 cybench.github.io，该网站链接 andyzorigin/cybench。在所审阅的 2026-09-24 提交中，仓库采用 Apache-2.0 许可证；运行需要模型 API 密钥、Python 3.9–3.10 以及使用 cybench/cybench 镜像的 Docker。其 task_list.txt 列出全部 40 道题目，但九道 GlacierCTF 题目（包括逆向题 RPGO 与 SOP）不在仓库目录中。", "pdf_url": "https://proceedings.iclr.cc/paper_files/paper/2025/file/3e9412a9c1d93810ef3ef7825115016b-Paper-Conference.pdf", "project_url": "https://cybench.github.io/", "publication_date": null, "publication_status": "conference-paper", "record_type": "paper", "runtime_tested": false, "scope_limitations_en": "Only 6 of the 40 tasks are reverse engineering; the rest are cryptography (16), web (8), forensics (4), miscellaneous (4) and pwn (2), so aggregate rates are not reverse-engineering accuracy, and Cybench is not a reverse-engineering-only benchmark. In the unguided structured-bash runs, six or seven of the eight models solved the two easiest reversing tasks (first human solves in 2 and 6 minutes), and none solved the other four. Results cover 2024 models (GPT-4o, o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 and 3.1, Mixtral 8x22b) with the authors' scaffolds: 15 iterations unguided or 5 per subtask, short memory, no decompiler tools, one attempt in the main comparison and the best of three in the scaffold comparison. First-solve times come from competition records and are not directly comparable across competitions. The authors judge train–test overlap by model knowledge cutoffs and cannot rule it out for Claude 3.5 Sonnet. CTF tasks are short, small-codebase competition problems, and capturing a flag does not show recovered program structure.", "scope_limitations_zh": "40 道题目中只有 6 道属于逆向工程，其余为密码学（16）、Web（8）、取证（4）、杂项（4）与漏洞利用（2），因此总体成功率并不代表逆向工程准确率，Cybench 也不是专门的逆向工程基准。在无引导的结构化 bash 实验中，八个模型中有六到七个解出了两道最简单的逆向题（人类首解用时分别为 2 分钟和 6 分钟），其余四道均无模型解出。结果仅涵盖 2024 年的模型（GPT-4o、o1-preview、Claude 3 Opus、Claude 3.5 Sonnet、Gemini 1.5 Pro、Llama 3 与 3.1、Mixtral 8x22b）及作者的智能体框架：无引导模式 15 轮、子任务模式每个子任务 5 轮，记忆较短，未提供反编译器工具；主比较只尝试一次，框架比较取三次尝试中的最好结果。首解用时来自各竞赛记录，不同竞赛之间不能直接比较。作者依据模型知识截止日期判断训练—测试重叠，无法排除 Claude 3.5 Sonnet 存在重叠。夺旗赛题目是用时较短、代码规模较小的竞赛问题，夺得旗标并不表明恢复了程序结构。", "targets": [ "binary" ], "title": "Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models", "title_zh": "Cybench：评估语言模型网络安全能力与风险的框架", "url": "https://proceedings.iclr.cc/paper_files/paper/2025/hash/3e9412a9c1d93810ef3ef7825115016b-Abstract-Conference.html", "venue": "ICLR 2025", "verification": "source-reviewed", "verified_at": "2026-10-19", "why_fits_en": "Adds a CTF agent benchmark with reverse-engineering tasks and subtask-level grading; EnIGMA in this section reports results on it, and it complements the reverse-engineering-specific benchmarks here.", "why_fits_zh": "补充一个含逆向工程题目并按子任务评分的夺旗赛智能体基准；本节收录的 EnIGMA 也报告了在该基准上的结果，它与本节专门面向逆向工程的基准互补。", "year": 2025 }