{ "added_at": "2026-10-19", "arxiv_id": "2409.16165", "arxiv_version_reviewed": "v3", "authors": [ "Talor Abramovich", "Meet Udeshi", "Minghao Shao", "Kilian Lieret", "Haoran Xi", "Kimberly Milner", "Sofija Jancheska", "John Yang", "Carlos E Jimenez", "Farshad Khorrami", "Prashanth Krishnamurthy", "Brendan Dolan-Gavitt", "Muhammad Shafique", "Karthik R Narasimhan", "Ramesh Karri", "Ofir Press" ], "bibtex_url": "https://awesomerea.org/papers/citations/paper-enigma.bib", "code_url": "https://github.com/SWE-agent/SWE-agent/tree/v0.7", "date_basis": "PMLR states that Volume 267 was published on 06 October 2025, matching the paper page’s citation_publication_date of 2025/10/06. The 13–19 July 2025 conference dates are not used.", "date_basis_zh": "PMLR 说明第 267 卷于 2025 年 10 月 6 日出版，与论文页面 citation_publication_date 所载 2025/10/06 一致；本条目不采用 2025 年 7 月 13–19 日的会议日期。", "date_precision": "day", "description_en": "EnIGMA extends SWE-agent for capture-the-flag challenges with interactive agent tools that keep a gdb debugging session or a pwntools server connection open across turns, alongside decompile and disassemble commands, an LM summarizer for long outputs, and category-specific demonstrations and guidelines. It is evaluated with four LMs on 390 challenges from NYU CTF, InterCode-CTF, Cybench and a HackTheBox set the authors collected. The paper ablates tools, summarizers and demonstrations, and analyzes solution leakage and “soliloquizing”, where the model writes its own observations instead of acting in the environment.", "description_zh": "EnIGMA 在 SWE-agent 基础上面向 CTF 题目进行扩展：交互式智能体工具可在多轮之间保持 gdb 调试会话或 pwntools 服务器连接，另有反编译与反汇编命令、用于长输出的语言模型摘要器，以及按题目类别编写的示例和指导。论文使用四种语言模型，在 NYU CTF、InterCode-CTF、Cybench 及作者收集的 HackTheBox 题集共 390 道题目上评测；还对工具、摘要器和示例做消融实验，并分析解答泄露和“自言自语”（soliloquizing）现象，即模型不与环境交互而自行编写观察结果。", "doi": null, "entry_url": "https://awesomerea.org/papers/paper-enigma/", "entry_url_zh": "https://awesomerea.org/zh/papers/paper-enigma/", "evidence_urls": [ "https://proceedings.mlr.press/v267/abramovich25a.html", "https://raw.githubusercontent.com/mlresearch/v267/main/assets/abramovich25a/abramovich25a.pdf", "https://proceedings.mlr.press/v267/", "https://arxiv.org/abs/2409.16165", "https://arxiv.org/abs/2409.16165v3", "https://enigma-agent.com/", "https://github.com/SWE-agent/SWE-agent/blob/084d5d164729ddcbe8fbc05877ead403549ef669/README.md", "https://github.com/SWE-agent/SWE-agent/blob/084d5d164729ddcbe8fbc05877ead403549ef669/LICENSE", "https://github.com/SWE-agent/SWE-agent/blob/084d5d164729ddcbe8fbc05877ead403549ef669/docs/usage/enigma.md", "https://github.com/SWE-agent/SWE-agent/blob/3ea751c087f32b16e039a2233dd6eefecef325d5/README.md", "https://github.com/enigma-agent/trajectories/blob/431dda6896b261ecebf658a5e6fc05ee695efe76/LICENSE", "https://github.com/enigma-agent/benchmarks/blob/c632a3c5d08f5f35eedc57f8c7a57be78defc013/README.md" ], "first_arxiv_date": "2024-09-24", "id": "paper-enigma", "json_url": "https://awesomerea.org/papers/paper-enigma/record.json", "kind": "research", "markdown_url": "https://awesomerea.org/papers/paper-enigma/index.md", "markdown_url_zh": "https://awesomerea.org/zh/papers/paper-enigma/index.md", "metadata_notes": "The 16-author order follows the PMLR page and the arXiv record; the PMLR metadata writes ‘Carlos E Jimenez’ and ‘Karthik R Narasimhan’, while the PDF prints ‘Carlos E. Jimenez’ and ‘Karthik Narasimhan’. PMLR lists pages 246–355 and no DOI; the arXiv v3 comment reads ‘ICML 2025’. The SWE-agent README still cites the earlier arXiv title ‘EnIGMA: Enhanced Interactive Generative Model Agent for CTF Challenges’. PMLR’s software link is the moving v0.7 branch; head 084d5d1 (2025-04-28) is MIT-licensed and documents running challenges with Docker image sweagent/enigma:latest. The current main README says to use SWE-agent 0.7 while EnIGMA is updated for 1.0. The project site links trajectories (MIT); the separate benchmarks repository reformats Cybench and InterCode-CTF only and leaves their copyright with the original creators, so it does not supply the HackTheBox set. Table 9 reports the NYU-agent overall rates as 3.00 and 4.00 in the reverse order of the main text.", "metadata_notes_zh": "16 位作者的顺序以 PMLR 页面和 arXiv 记录为准；PMLR 元数据写作“Carlos E Jimenez”和“Karthik R Narasimhan”，PDF 印作“Carlos E. Jimenez”和“Karthik Narasimhan”。PMLR 列出页码 246–355，未提供 DOI；arXiv v3 备注为“ICML 2025”。SWE-agent README 仍引用较早的 arXiv 标题“EnIGMA: Enhanced Interactive Generative Model Agent for CTF Challenges”。PMLR 的软件链接指向会变动的 v0.7 分支；分支头 084d5d1（2025-04-28）采用 MIT 许可证，并说明使用 Docker 镜像 sweagent/enigma:latest 运行题目。当前 main 分支 README 称，在 EnIGMA 更新至 1.0 期间应使用 SWE-agent 0.7。项目网站链接了运行轨迹（MIT）；另一个基准仓库仅重新格式化了 Cybench 与 InterCode-CTF，版权仍归原作者，因此并未提供 HackTheBox 题集。表 9 中 NYU 智能体的总体解题率 3.00 与 4.00 与正文所述顺序相反。", "pdf_url": "https://raw.githubusercontent.com/mlresearch/v267/main/assets/abramovich25a/abramovich25a.pdf", "project_url": "https://enigma-agent.com/", "publication_date": "2025-10-06", "publication_status": "conference-paper", "record_type": "paper", "runtime_tested": false, "scope_limitations_en": "The benchmarks mix cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks (104 of the 390 are reverse engineering), so aggregate solve rates such as 13.5% on NYU CTF are not reverse-engineering accuracy. The per-category table reports 17.65% on NYU CTF’s 51 reverse-engineering challenges with Claude 3.5 Sonnet. Results are pass@1 with a $3 per-instance cost cap and 2024 models (GPT-4 Turbo, GPT-4o, Claude 3.5 Sonnet and LLaMA 3.1 405B Instruct); the NYU CTF baseline comparison uses pass@5. Many challenges have public write-ups, and the authors’ leakage and soliloquy measures are heuristics; on 21 later CSAW 2024 challenges the best configurations solved 2. Solving a CTF shows a captured flag, not recovered program structure or source.", "scope_limitations_zh": "各基准混合了密码学、取证、二进制漏洞利用、逆向工程、Web 与杂项任务（390 道题中有 104 道为逆向工程），因此 NYU CTF 上 13.5% 等总体解题率并不等于逆向工程准确率。分类别表格报告，使用 Claude 3.5 Sonnet 时，NYU CTF 中 51 道逆向工程题的解题率为 17.65%。结果采用 pass@1，每道题成本上限为 3 美元，使用 2024 年的模型（GPT-4 Turbo、GPT-4o、Claude 3.5 Sonnet 与 LLaMA 3.1 405B Instruct）；NYU CTF 基线对比采用 pass@5。许多题目存在公开题解，作者对泄露和“自言自语”的度量属于启发式方法；在 21 道较晚发布的 CSAW 2024 题目上，最佳配置解出 2 道。解出 CTF 题目表明获得了 flag，而非恢复了程序结构或源码。", "targets": [ "binary" ], "title": "EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities", "title_zh": "EnIGMA：交互式工具显著帮助语言模型智能体发现安全漏洞", "url": "https://proceedings.mlr.press/v267/abramovich25a.html", "venue": "ICML 2025", "verification": "source-reviewed", "verified_at": "2026-10-18", "why_fits_en": "Evaluates how interactive debugger and connection tools change agent performance on CTF tasks that include reverse engineering, complementing the reverse-engineering-specific benchmarks in this section.", "why_fits_zh": "评估交互式调试和连接工具如何影响智能体在包含逆向工程的 CTF 任务上的表现，与本节中专门针对逆向工程的基准互补。", "year": 2025 }