# Large Language Models for Code Analysis: Do LLMs Really Do Their Job? > Evaluates GPT-3.5-turbo, GPT-4, LLaMA-2-13B, Code-LLaMA-2-13B-Instruct and StarChat-Beta on explaining what de-commented JavaScript, Python and C code does, then repeats the task on JavaScript obfuscated with JavaScript Obfuscator (identifier renaming, dead-code injection, control-flow flattening and string splitting) and with Wobfuscator’s WebAssembly transformation. GPT-3.5 and GPT-4 are also asked to rewrite 100 IOCCC winning programs as readable, compilable code. Case studies apply GPT models to a botnet and a benign browser-extension repository, an Android SMS stealer decompiled with JD-GUI, and WannaCry code decompiled with RetDec. - Authors: Chongzhou Fang; Ning Miao; Shaurya Srivastav; Jialin Liu; Ruoyu Zhang; Ruijie Fang; Asmita; Ryan Tsang; Najmeh Nazari; Han Wang; Houman Homayoun - Venue: USENIX Security 2024 - Status: Conference paper - Publication / posting date: August 2024 - Date source: Publication month in the official USENIX proceedings BibTeX; the August 14–16, 2024 symposium dates are not used to assert a publication day. - First arXiv submission: October 18, 2023 - arXiv: 2310.12357 - Reviewed arXiv version: v2 - Sources reviewed: 2026-10-18 - Added to the catalog: 2026-10-19 - Original paper: - PDF: - Project: - BibTeX: - Catalog page: - Record JSON: - Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-llm-code-analysis/index.md) ## Scope and limitations Ground truth is GPT-4’s own explanations after manual checking by four students; other models are scored against those texts with similarity metrics and a GPT-4 judge, which the authors say is unreliable without manual calibration. Correctness is judged at explanation level, not by recovered behavior or executable equivalence. Only JavaScript is obfuscated with generated transformations; neither GPT model deciphered the WebAssembly that Wobfuscator inserts, and the smaller models produced no meaningful explanations once obfuscation was applied. Author-reported figures include 97.4% GPT-4 accuracy on non-obfuscated code and 87% on obfuscated JavaScript; in IOCCC de-obfuscation, about 20% of GPT-3.5’s targets and 19% of GPT-4’s yielded compilable code, and only 8% and 4% produced correct results. The malware analyses are qualitative case studies on a few samples, with manual chunking for WannaCry, missed behavior when context was split, and a false alarm in the botnet repository. Models recognized code from popular repositories, so memorization is a confound. The evaluated GPT-3.5/GPT-4-era and LLaMA-2-13B-class models are dated. The study analyzes source or already-decompiled text; it does not perform binary decompilation. ## Why it fits this collection Measures how far LLM explanations survive JavaScript obfuscation and decompiler output, complementing JavaScript deobfuscation benchmarks and tools in the collection. ## Publication and review notes USENIX publishes no DOI for the paper; the arXiv identifier is recorded, and the final PDF name usenixsecurity24-fang.pdf does not use the secYY publisher profile. The USENIX page, its BibTeX and the final PDF agree on the 11-author order, the single-name author Asmita, pages 829–846 and ISBN 978-1-939133-44-1; the page also lists a prepublication PDF. arXiv v2 (2024-03-05) carries the comment “Accepted by Usenix Security 2024”. The paper names the online appendix; at the reviewed 2024-08-28 commit it holds dataset, result and case-study folders and declares no license, and its samples include third-party code that keeps its own terms. Model versions beyond family names are not specified in the paper. ## Primary sources 1. 2. 3. 4. 5. 6. ## About this entry Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX. Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.