# How Far Have We Gone in Binary Code Understanding Using Large Language Models

> Benchmarks 16 code and general LLMs on function name recovery and summarization of stripped binaries. The benchmark takes 12 popular C projects built for x86-64, strips and decompiles 28 binaries with IDA Pro, aligns 81,185 functions with source through DWARF, and keeps 2,000 functions after expert review. Zero-shot LLMs are compared with SymLM, NER, BinT5 and HexT5. Code Llama 34B leads name recovery with token-level F1 of 27.59, and ChatGPT leads summarization with BLEU-4 7.37. The study also measures few-shot prompting, LoRA fine-tuning of two 7B models on GNU binaries, and a case study on a stripped open-source Linux remote-access trojan.

- Authors: Xiuwei Shang; Shaoyin Cheng; Guoqiang Chen; Yanming Zhang; Li Hu; Xiao Yu; Gangyang Li; Weiming Zhang; Nenghai Yu
- Venue: ICSME 2024
- Status: Conference paper
- Publication / posting date: October 6, 2024
- Date source: IEEE’s Crossref deposit gives published/print and issued date 2024-10-06, which is also the first day of the conference listed there (6–11 October). This entry uses the publisher-deposited date and does not derive it from the conference schedule or the arXiv history.
- First arXiv submission: April 15, 2024
- arXiv: 2404.09836
- Reviewed arXiv version: v3
- DOI: 10.1109/ICSME58944.2024.00012
- Sources reviewed: 2026-10-20
- Added to the catalog: 2026-11-03
- Original paper: <https://ieeexplore.ieee.org/document/10795058>
- PDF: <https://arxiv.org/pdf/2404.09836v3>
- BibTeX: <https://awesomerea.org/papers/citations/paper-llm-binary-understanding.bib>
- Catalog page: <https://awesomerea.org/papers/paper-llm-binary-understanding/>
- Record JSON: <https://awesomerea.org/papers/paper-llm-binary-understanding/record.json>
- Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-llm-binary-understanding/index.md)

## Scope and limitations

The benchmark covers x86-64 builds with each project’s default compiler and optimization settings and IDA Pro pseudocode only, so the results do not establish behavior on other architectures, optimization levels, decompilers or obfuscated code, which the authors exclude. Reference summaries were generated by ChatGPT from source and screened by three experts, and ChatGPT is also an evaluated model. BLEU-4, METEOR and Rouge-L measure overlap with those references, which the authors say may not fit binary summarization. Name recovery uses token-level precision, recall and F1 rather than exact matches. Fine-tuning used 30,000 functions from O0 x86-64 GNU builds. Results describe model versions released between mid-2023 and early 2024, including gpt-3.5-turbo-16k.

## Why it fits this collection

Adds a benchmark of LLMs on two core reverse-engineering subtasks, recovering function names and summarizing decompiled functions, with comparisons against earlier learned symbol-recovery models.

## Publication and review notes

Crossref establishes the nine-author order, DOI, ICSME 2024 proceedings title and pages 1–12. Reviewed arXiv v3 (2024-10-24), whose comment reads “to be published in ICSME 2024”; v1 and v2 date from 15 and 16 April 2024. The IEEE Xplore page refused automated requests. The paper gives no download link for the benchmark, fine-tuning data or prompts, and no release was found. The evaluated pseudocode depends on IDA Pro, a commercial decompiler.

## Primary sources

1. <https://api.crossref.org/works/10.1109/ICSME58944.2024.00012>
2. <https://arxiv.org/abs/2404.09836v3>
3. <https://arxiv.org/pdf/2404.09836v3>
4. <https://arxiv.org/html/2404.09836v3>

## About this entry

Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.