{ "added_at": "2026-10-27", "arxiv_id": "2406.05590", "arxiv_version_reviewed": "v3", "authors": [ "Minghao Shao", "Sofija Jancheska", "Meet Udeshi", "Brendan Dolan-Gavitt", "Haoran Xi", "Kimberly Milner", "Boyuan Chen", "Max Yin", "Siddharth Garg", "Prashanth Krishnamurthy", "Farshad Khorrami", "Ramesh Karri", "Muhammad Shafique" ], "bibtex_url": "https://awesomerea.org/papers/citations/paper-nyu-ctf-bench.bib", "code_url": "https://github.com/NYU-LLM-CTF/NYU_CTF_Bench", "date_basis": "The official NeurIPS proceedings and publisher-deposited Crossref metadata identify publication year 2024; the proceedings page’s citation_publication_date metadata reads 2024-12-16, but neither Crossref nor a publisher statement confirms that day, so only the year is recorded. The first arXiv submission date is recorded separately.", "date_basis_zh": "NeurIPS 官方论文集与出版方提交的 Crossref 元数据标明出版年份为 2024；论文集页面的 citation_publication_date 元数据为 2024-12-16，但 Crossref 与出版方说明均未确认该日期，因此仅记录年份。首次提交 arXiv 的日期另行记录。", "date_precision": "year", "description_en": "NYU CTF Bench packages 200 manually validated challenges from CSAW CTF qualifying and final rounds held between 2017 and 2023, spanning cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks, with challenge.json metadata and Docker images for server-side challenges. The paper also presents an automated framework that gives LLMs function-calling tools for shell commands, file creation, Ghidra disassembly and decompilation, flag checking and giving up. It evaluates five LLMs and compares their scores with human teams at CSAW 2022 and 2023.", "description_zh": "NYU CTF Bench 收录 2017–2023 年 CSAW CTF 资格赛与决赛中经人工验证的 200 道题目，涵盖密码学、取证、二进制漏洞利用、逆向工程、Web 与杂项任务，并为题目提供 challenge.json 元数据，为需要服务端的题目提供 Docker 镜像。论文还提出一个自动化框架，通过函数调用为大语言模型提供执行 Shell 命令、创建文件、使用 Ghidra 反汇编与反编译、检查 flag 和放弃求解等工具。论文评测了五种大语言模型，并将其得分与 CSAW 2022 和 2023 的人类参赛队伍比较。", "doi": "10.52202/079017-1832", "entry_url": "https://awesomerea.org/papers/paper-nyu-ctf-bench/", "entry_url_zh": "https://awesomerea.org/zh/papers/paper-nyu-ctf-bench/", "evidence_urls": [ "https://proceedings.neurips.cc/paper_files/paper/2024/hash/69d97a6493fbf016fff0a751f253ad18-Abstract-Datasets_and_Benchmarks_Track.html", "https://proceedings.neurips.cc/paper_files/paper/2024/file/69d97a6493fbf016fff0a751f253ad18-Paper-Datasets_and_Benchmarks_Track.pdf", "https://api.crossref.org/works/10.52202/079017-1832", "https://arxiv.org/abs/2406.05590", "https://arxiv.org/pdf/2406.05590v3", "https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/blob/4c5744f55a31a58e03d6118c9dbb1c18e5bbdfa9/README.md", "https://github.com/NYU-LLM-CTF/NYU_CTF_Bench/blob/4c5744f55a31a58e03d6118c9dbb1c18e5bbdfa9/LICENSE", "https://github.com/NYU-LLM-CTF/nyuctf_agents/blob/612190f298ae9e604f00370019beed4ba1f372f6/README.md", "https://zenodo.org/records/13930622" ], "first_arxiv_date": "2024-06-08", "id": "paper-nyu-ctf-bench", "json_url": "https://awesomerea.org/papers/paper-nyu-ctf-bench/record.json", "kind": "research", "markdown_url": "https://awesomerea.org/papers/paper-nyu-ctf-bench/index.md", "markdown_url_zh": "https://awesomerea.org/zh/papers/paper-nyu-ctf-bench/index.md", "metadata_notes": "The 13-author order follows the NeurIPS proceedings page, the PDF, arXiv and Crossref. NeurIPS volume 37 lists pages 57472–57498; arXiv v3 (2025-02-18) matches the reviewed proceedings text. The NeurIPS checklist gives the Zenodo software DOI 10.5281/zenodo.13930622 for the v20241008 benchmark release; it is an artifact identifier, not the paper DOI. The reviewed NYU_CTF_Bench snapshot (2025-09-22) is GPL-2.0, holds the 200-challenge test split and a 55-challenge development split, and is installed through the nyuctf Python package, which clones the repository; challenges need Docker. The paper’s llm_ctf_automation link now redirects to NYU-LLM-CTF/nyuctf_agents, whose reviewed 2026-07-17 snapshot is MIT-licensed and also contains the later D-CIPHER multi-agent system alongside the baseline. Challenge files keep their original authors’ terms.", "metadata_notes_zh": "13 位作者的顺序以 NeurIPS 论文集页面、PDF、arXiv 与 Crossref 为准。NeurIPS 第 37 卷列出页码 57472–57498；arXiv v3（2025-02-18）与所审阅的论文集文本一致。NeurIPS 检查清单给出 Zenodo 软件 DOI 10.5281/zenodo.13930622，对应 v20241008 基准发布版本；它是研究产物标识符，并非论文 DOI。所审阅的 NYU_CTF_Bench 快照（2025-09-22）采用 GPL-2.0，包含 200 道题的测试集和 55 道题的开发集，通过会克隆该仓库的 nyuctf Python 包安装，题目需要 Docker。论文中的 llm_ctf_automation 链接现重定向至 NYU-LLM-CTF/nyuctf_agents；其所审阅的 2026-07-17 快照采用 MIT 许可证，除基线外还包含后来的 D-CIPHER 多智能体系统。题目文件沿用原作者的条款。", "pdf_url": "https://proceedings.neurips.cc/paper_files/paper/2024/file/69d97a6493fbf016fff0a751f253ad18-Paper-Datasets_and_Benchmarks_Track.pdf", "project_url": "https://nyu-llm-ctf.github.io/", "publication_date": null, "publication_status": "conference-paper", "record_type": "paper", "runtime_tested": false, "scope_limitations_en": "Only 51 of the 200 challenges are reverse engineering (31 qualifying and 20 final), so overall solve rates are not reverse-engineering accuracy. All challenges come from one competition series, categories are imbalanced, and the remaining 367 of the initial 567 challenges, including incident-response tasks, are excluded pending validation. The baseline counts a challenge as solved if any of five attempts succeeds, within a 48-hour limit per model; no statistical tests were run. Among GPT-3.5, GPT-4, Claude 3, Mixtral and LLaMA, the best reverse-engineering rate was 9.80% (GPT-4), and the open models solved nothing. The human comparison uses competition scores from four CSAW events, and the public repository has continued to change after publication, so later runs may not match the paper’s challenge set or framework.", "scope_limitations_zh": "200 道题中只有 51 道为逆向工程题（资格赛 31 道、决赛 20 道），因此总体解题率并不等于逆向工程准确率。所有题目均来自同一系列赛事，各类别数量不均衡；最初 567 道题中其余 367 道（包括事件响应类任务）因尚未完成验证而未收录。基线评测中，五次尝试中任意一次成功即算解出，每个模型限时 48 小时，且未进行统计检验。在 GPT-3.5、GPT-4、Claude 3、Mixtral 与 LLaMA 中，逆向工程类最佳解题率为 9.80%（GPT-4），两个开源模型未解出任何题目。与人类的比较基于四场 CSAW 赛事的比赛得分；公开仓库在论文发表后仍持续变化，后续运行未必与论文中的题目集或框架一致。", "targets": [ "binary" ], "title": "NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security", "title_zh": "NYU CTF Bench：用于评估大语言模型攻击性安全能力的可扩展开源基准数据集", "url": "https://proceedings.neurips.cc/paper_files/paper/2024/hash/69d97a6493fbf016fff0a751f253ad18-Abstract-Datasets_and_Benchmarks_Track.html", "venue": "NeurIPS 2024 Datasets and Benchmarks", "verification": "source-reviewed", "verified_at": "2026-10-18", "why_fits_en": "Supplies a public, dockerized CTF challenge set with a reverse-engineering subset; later agent work such as EnIGMA evaluates on it.", "why_fits_zh": "提供一个容器化的公开 CTF 题目集，其中包含逆向工程子集；EnIGMA 等后续智能体研究在其上开展评测。", "year": 2024 }