# BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models > BinMetric poses 1,000 questions across six binary-analysis tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification and assembly instruction generation. The questions come from 20 open-source C projects compiled for x86-64, stripped and processed with IDA Pro, and aligned with source code through DWARF information. The authors evaluate 12 LLMs from five model families with one-shot prompts, alongside IDA Pro’s decompiler, LLM4Decompile, BinT5 and HexT5 where those apply. - Authors: Xiuwei Shang; Guoqiang Chen; Shaoyin Cheng; Benlong Wu; Li Hu; Gangyang Li; Weiming Zhang; Nenghai Yu - Venue: IJCAI 2025 - Status: Conference paper - Publication / posting date: September 2025 - Date source: The IJCAI Organization’s Crossref deposit gives publication month September 2025. The proceedings page’s citation metadata separately gives 2025-09-16; this entry keeps the month that both sources state and does not use the August 16–22 conference dates. - First arXiv submission: May 12, 2025 - arXiv: 2505.07360 - Reviewed arXiv version: v1 - DOI: 10.24963/ijcai.2025/858 - Sources reviewed: 2026-10-18 - Added to the catalog: 2026-10-19 - Original paper: - PDF: - BibTeX: - Catalog page: - Record JSON: - Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-binmetric/index.md) ## Scope and limitations Scores are question-level metrics on isolated snippets, not end-to-end reverse-engineering workflows or agentic tool use. Preprocessing requires IDA Pro, and all binaries are x86-64 builds with each project’s default compiler settings on Ubuntu 22.04; other architectures, optimization sweeps and obfuscation are not studied. Most tasks score text overlap with Rouge-L, BLEU, METEOR or CodeBLEU, so they do not test semantic equivalence; only assembly generation checks compilation and test-case execution, where the average execution pass rate is below 1%. Summary labels were generated by ChatGPT and then manually reviewed, and algorithm classification uses 80 snippets from a single algorithms project. Leakage screening relied on exact-match web searches. The evaluated models predate 2025 (including gpt-3.5-turbo-16k-0613 and gpt-4-0613) and ran with a maximum length of 8,192 tokens, so the rankings are historical. ## Why it fits this collection Adds a multi-task benchmark spanning binary lifting, semantic recovery, algorithm identification and assembly synthesis, complementing single-task decompilation and agent benchmarks. ## Publication and review notes The proceedings title reads “Binary Code Analysis Benchmark”; arXiv v1, the only version, is titled “A Comprehensive Binary Analysis Benchmark for Large Language Models” and is a 23-page extended version whose comment says it was to be published in IJCAI 2025. The proceedings PDF, page metadata and Crossref agree on the eight-author order, the DOI and pages 7715–7723 in the Main Track. Neither the proceedings PDF nor arXiv v1 links code, data or a leaderboard. A same-named GitHub repository describes itself as the IJCAI’25 benchmark but declares no license and is not linked from the paper. Per-task counts are 70 call-site, 250 decompilation, 250 signature, 250 summarization, 80 classification and 100 assembly-generation items. ## Primary sources 1. 2. 3. 4. 5. ## About this entry Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX. Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.