{ "added_at": "2026-10-19", "arxiv_id": "2505.07360", "arxiv_version_reviewed": "v1", "authors": [ "Xiuwei Shang", "Guoqiang Chen", "Shaoyin Cheng", "Benlong Wu", "Li Hu", "Gangyang Li", "Weiming Zhang", "Nenghai Yu" ], "bibtex_url": "https://awesomerea.org/papers/citations/paper-binmetric.bib", "code_url": null, "date_basis": "The IJCAI Organization’s Crossref deposit gives publication month September 2025. The proceedings page’s citation metadata separately gives 2025-09-16; this entry keeps the month that both sources state and does not use the August 16–22 conference dates.", "date_basis_zh": "IJCAI 组织向 Crossref 提交的元数据给出出版月份为 2025 年 9 月；论文集页面的引用元数据另给出 2025-09-16。本条目采用两处来源均载明的月份，不使用 8 月 16–22 日的会议日期。", "date_precision": "month", "description_en": "BinMetric poses 1,000 questions across six binary-analysis tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification and assembly instruction generation. The questions come from 20 open-source C projects compiled for x86-64, stripped and processed with IDA Pro, and aligned with source code through DWARF information. The authors evaluate 12 LLMs from five model families with one-shot prompts, alongside IDA Pro’s decompiler, LLM4Decompile, BinT5 and HexT5 where those apply.", "description_zh": "BinMetric 提供六类二进制分析任务共 1,000 道题目：调用点重建、反编译、函数签名恢复、二进制代码摘要、算法分类与汇编指令生成。题目取自 20 个开源 C 项目：项目针对 x86-64 编译、剥离符号并经 IDA Pro 处理，再借助 DWARF 信息与源码对齐。作者使用单样例提示评测来自五个模型家族的 12 个大语言模型，并在适用任务上与 IDA Pro 反编译器、LLM4Decompile、BinT5 和 HexT5 对比。", "doi": "10.24963/ijcai.2025/858", "entry_url": "https://awesomerea.org/papers/paper-binmetric/", "entry_url_zh": "https://awesomerea.org/zh/papers/paper-binmetric/", "evidence_urls": [ "https://www.ijcai.org/proceedings/2025/858", "https://www.ijcai.org/proceedings/2025/0858.pdf", "https://api.crossref.org/works/10.24963/ijcai.2025/858", "https://arxiv.org/abs/2505.07360", "https://arxiv.org/pdf/2505.07360v1" ], "first_arxiv_date": "2025-05-12", "id": "paper-binmetric", "json_url": "https://awesomerea.org/papers/paper-binmetric/record.json", "kind": "research", "markdown_url": "https://awesomerea.org/papers/paper-binmetric/index.md", "markdown_url_zh": "https://awesomerea.org/zh/papers/paper-binmetric/index.md", "metadata_notes": "The proceedings title reads “Binary Code Analysis Benchmark”; arXiv v1, the only version, is titled “A Comprehensive Binary Analysis Benchmark for Large Language Models” and is a 23-page extended version whose comment says it was to be published in IJCAI 2025. The proceedings PDF, page metadata and Crossref agree on the eight-author order, the DOI and pages 7715–7723 in the Main Track. Neither the proceedings PDF nor arXiv v1 links code, data or a leaderboard. A same-named GitHub repository describes itself as the IJCAI’25 benchmark but declares no license and is not linked from the paper. Per-task counts are 70 call-site, 250 decompilation, 250 signature, 250 summarization, 80 classification and 100 assembly-generation items.", "metadata_notes_zh": "论文集中的标题为“Binary Code Analysis Benchmark”；arXiv 唯一版本 v1 的标题为“A Comprehensive Binary Analysis Benchmark for Large Language Models”，是 23 页的扩展版本，备注称将发表于 IJCAI 2025。论文集 PDF、页面元数据与 Crossref 一致给出八位作者顺序、DOI 及主会第 7715–7723 页。论文集 PDF 与 arXiv v1 均未链接代码、数据或排行榜。GitHub 上一个同名仓库自称为该 IJCAI’25 基准，但未声明许可证，论文也未链接它。各任务题目数分别为调用点重建 70 道、反编译 250 道、签名恢复 250 道、代码摘要 250 道、算法分类 80 道、汇编生成 100 道。", "pdf_url": "https://www.ijcai.org/proceedings/2025/0858.pdf", "project_url": null, "publication_date": "2025-09", "publication_status": "conference-paper", "record_type": "paper", "runtime_tested": false, "scope_limitations_en": "Scores are question-level metrics on isolated snippets, not end-to-end reverse-engineering workflows or agentic tool use. Preprocessing requires IDA Pro, and all binaries are x86-64 builds with each project’s default compiler settings on Ubuntu 22.04; other architectures, optimization sweeps and obfuscation are not studied. Most tasks score text overlap with Rouge-L, BLEU, METEOR or CodeBLEU, so they do not test semantic equivalence; only assembly generation checks compilation and test-case execution, where the average execution pass rate is below 1%. Summary labels were generated by ChatGPT and then manually reviewed, and algorithm classification uses 80 snippets from a single algorithms project. Leakage screening relied on exact-match web searches. The evaluated models predate 2025 (including gpt-3.5-turbo-16k-0613 and gpt-4-0613) and ran with a maximum length of 8,192 tokens, so the rankings are historical.", "scope_limitations_zh": "得分是针对孤立代码片段的题目级指标，并不代表端到端逆向工程流程或智能体工具使用能力。预处理需要 IDA Pro，所有二进制均是在 Ubuntu 22.04 上以各项目默认编译设置构建的 x86-64 程序；未研究其他架构、优化级别变化或混淆。多数任务以 Rouge-L、BLEU、METEOR 或 CodeBLEU 衡量文本重合度，因而并不检验语义等价；只有汇编生成任务检查能否编译及测试用例执行结果，其平均执行通过率低于 1%。摘要标签由 ChatGPT 生成后经人工审核；算法分类使用取自单一算法项目的 80 个片段。数据泄露筛查依赖网页精确匹配搜索。被评测模型均早于 2025 年（包括 gpt-3.5-turbo-16k-0613 与 gpt-4-0613），最大长度设为 8,192 个词元，因此排名只反映当时情况。", "targets": [ "binary" ], "title": "BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models", "title_zh": "BinMetric：面向大语言模型的综合二进制代码分析基准", "url": "https://www.ijcai.org/proceedings/2025/858", "venue": "IJCAI 2025", "verification": "source-reviewed", "verified_at": "2026-10-18", "why_fits_en": "Adds a multi-task benchmark spanning binary lifting, semantic recovery, algorithm identification and assembly synthesis, complementing single-task decompilation and agent benchmarks.", "why_fits_zh": "补充一个涵盖二进制提升、语义恢复、算法识别与汇编合成的多任务基准，与单一任务的反编译及智能体基准互补。", "year": 2025 }