2022 · Preprint
Pop Quiz! Can a Large Language Model Help With Reverse Engineering?
Tests whether OpenAI’s Codex model code-davinci-001 can answer reverse-engineering questions about C code in a zero-shot question-and-answer prompt. Programs come from templates the authors wrote, covering cybersecurity snippets and industrial-control algorithms such as PID controllers and filters, plus five single-file Linux malware samples from vx-underground. Each is presented as original or name-randomized source, or as Ghidra decompilation of gcc -O1 builds with debug information, without it, or stripped. After exploratory open-ended questions, generated true/false and short-answer questions about purpose, capabilities and key variables are graded against an answer oracle; Codex answered 72,754 of 136,260 questions correctly.
Scope and limitations
All results come from one model, code-davinci-001, which OpenAI shut down on 2023-03-23, so they cannot be rerun against the original model and do not describe current LLMs. Accuracy is measured with a generated true/false and short-answer quiz, paired as positive and negated questions to expose answer bias, rather than with analyst tasks; the overall 53.4% led the authors to conclude that LLMs are not yet ready for zero-shot reverse engineering. Programs are small single-file C code that fits the model’s roughly 4,000-token window, compiled only at -O1; the real-malware class has five samples. The answer oracle combines the authors’ labels with capability matches from the capa tool.
Why it fits this collection
Records the earliest LLM-for-reverse-engineering evaluation in the bibliography, a baseline for the later LLM binary-analysis studies in this section.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- February 2, 2022
- Date source
- arXiv v1 submission date; no later version, journal reference or proceedings publication was found.
- First arXiv submission
- February 2, 2022
- arXiv
- 2202.01142
- Reviewed arXiv version
- v1
Reviewed arXiv v1, the only version; its comment reads “18 pages, 19 figures” and links Zenodo record 10.5281/zenodo.5949075, a dataset DOI that is not used as the paper identifier. No peer-reviewed version was found in Crossref. That record (CC BY 4.0, a 692 MB codex-reverse.zip by the same six creators) holds the test framework scripts, recorded Codex responses and generated program instances. The README requires Ghidra and an OpenAI API key, and the build log lists compiled instances under experiments/malware, so the archive appears to include the malware samples.
Sources reviewed
Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.