{ "added_at": "2026-10-19", "arxiv_id": "2310.12357", "arxiv_version_reviewed": "v2", "authors": [ "Chongzhou Fang", "Ning Miao", "Shaurya Srivastav", "Jialin Liu", "Ruoyu Zhang", "Ruijie Fang", "Asmita", "Ryan Tsang", "Najmeh Nazari", "Han Wang", "Houman Homayoun" ], "bibtex_url": "https://awesomerea.org/papers/citations/paper-llm-code-analysis.bib", "code_url": null, "date_basis": "Publication month in the official USENIX proceedings BibTeX; the August 14–16, 2024 symposium dates are not used to assert a publication day.", "date_basis_zh": "采用 USENIX 官方论文集 BibTeX 中的出版月份，不以 2024 年 8 月 14–16 日的会议日期推定具体出版日。", "date_precision": "month", "description_en": "Evaluates GPT-3.5-turbo, GPT-4, LLaMA-2-13B, Code-LLaMA-2-13B-Instruct and StarChat-Beta on explaining what de-commented JavaScript, Python and C code does, then repeats the task on JavaScript obfuscated with JavaScript Obfuscator (identifier renaming, dead-code injection, control-flow flattening and string splitting) and with Wobfuscator’s WebAssembly transformation. GPT-3.5 and GPT-4 are also asked to rewrite 100 IOCCC winning programs as readable, compilable code. Case studies apply GPT models to a botnet and a benign browser-extension repository, an Android SMS stealer decompiled with JD-GUI, and WannaCry code decompiled with RetDec.", "description_zh": "评估 GPT-3.5-turbo、GPT-4、LLaMA-2-13B、Code-LLaMA-2-13B-Instruct 与 StarChat-Beta 解释已去除注释的 JavaScript、Python 和 C 代码功能的能力，再在经 JavaScript Obfuscator（标识符重命名、死代码注入、控制流平坦化与字符串拆分）及 Wobfuscator WebAssembly 变换混淆的 JavaScript 上重复该任务。研究还要求 GPT-3.5 与 GPT-4 将 100 个 IOCCC 获奖程序改写为可读、可编译的代码。案例研究将 GPT 模型用于一个僵尸网络仓库与一个正常浏览器扩展仓库、经 JD-GUI 反编译的 Android 短信窃取程序，以及经 RetDec 反编译的 WannaCry 代码。", "doi": null, "entry_url": "https://awesomerea.org/papers/paper-llm-code-analysis/", "entry_url_zh": "https://awesomerea.org/zh/papers/paper-llm-code-analysis/", "evidence_urls": [ "https://www.usenix.org/conference/usenixsecurity24/presentation/fang", "https://www.usenix.org/system/files/usenixsecurity24-fang.pdf", "https://arxiv.org/abs/2310.12357", "https://arxiv.org/pdf/2310.12357v2", "https://github.com/aseec-lab/llms-for-code-analysis/blob/f1bdb01f1da9143bda796c259f06c7364683a102/README.md", "https://github.com/aseec-lab/llms-for-code-analysis/tree/f1bdb01f1da9143bda796c259f06c7364683a102" ], "first_arxiv_date": "2023-10-18", "id": "paper-llm-code-analysis", "json_url": "https://awesomerea.org/papers/paper-llm-code-analysis/record.json", "kind": "research", "markdown_url": "https://awesomerea.org/papers/paper-llm-code-analysis/index.md", "markdown_url_zh": "https://awesomerea.org/zh/papers/paper-llm-code-analysis/index.md", "metadata_notes": "USENIX publishes no DOI for the paper; the arXiv identifier is recorded, and the final PDF name usenixsecurity24-fang.pdf does not use the secYY publisher profile. The USENIX page, its BibTeX and the final PDF agree on the 11-author order, the single-name author Asmita, pages 829–846 and ISBN 978-1-939133-44-1; the page also lists a prepublication PDF. arXiv v2 (2024-03-05) carries the comment “Accepted by Usenix Security 2024”. The paper names the online appendix; at the reviewed 2024-08-28 commit it holds dataset, result and case-study folders and declares no license, and its samples include third-party code that keeps its own terms. Model versions beyond family names are not specified in the paper.", "metadata_notes_zh": "USENIX 未为该论文提供 DOI；本条目记录 arXiv 标识符，最终 PDF 文件名 usenixsecurity24-fang.pdf 也不符合 secYY 出版方规则。USENIX 页面、其 BibTeX 与最终 PDF 一致给出 11 位作者顺序、单名作者 Asmita、第 829–846 页及 ISBN 978-1-939133-44-1；页面还提供一份预发表 PDF。arXiv v2（2024-03-05）备注为“Accepted by Usenix Security 2024”。论文点名了在线附录；在所审阅的 2024-08-28 提交中，附录包含数据集、结果与案例研究目录，但未声明许可证，其中的样本包括保留各自条款的第三方代码。论文除模型家族名称外未说明具体模型版本。", "pdf_url": "https://www.usenix.org/system/files/usenixsecurity24-fang.pdf", "project_url": "https://github.com/aseec-lab/llms-for-code-analysis", "publication_date": "2024-08", "publication_status": "conference-paper", "record_type": "paper", "runtime_tested": false, "scope_limitations_en": "Ground truth is GPT-4’s own explanations after manual checking by four students; other models are scored against those texts with similarity metrics and a GPT-4 judge, which the authors say is unreliable without manual calibration. Correctness is judged at explanation level, not by recovered behavior or executable equivalence. Only JavaScript is obfuscated with generated transformations; neither GPT model deciphered the WebAssembly that Wobfuscator inserts, and the smaller models produced no meaningful explanations once obfuscation was applied. Author-reported figures include 97.4% GPT-4 accuracy on non-obfuscated code and 87% on obfuscated JavaScript; in IOCCC de-obfuscation, about 20% of GPT-3.5’s targets and 19% of GPT-4’s yielded compilable code, and only 8% and 4% produced correct results. The malware analyses are qualitative case studies on a few samples, with manual chunking for WannaCry, missed behavior when context was split, and a false alarm in the botnet repository. Models recognized code from popular repositories, so memorization is a confound. The evaluated GPT-3.5/GPT-4-era and LLaMA-2-13B-class models are dated. The study analyzes source or already-decompiled text; it does not perform binary decompilation.", "scope_limitations_zh": "真实标签是 GPT-4 自身生成、经四名学生人工核查的解释；其他模型则通过相似度指标与 GPT-4 评判与这些文本比较，作者也指出该评判方法若无人工校准并不可靠。正确性在解释层面判断，并非依据恢复出的行为或可执行等价性。只有 JavaScript 经过工具生成的混淆变换；两个 GPT 模型都未能解读 Wobfuscator 插入的 WebAssembly，较小模型在施加混淆后则无法给出有意义的解释。作者报告 GPT-4 在未混淆代码上的准确率为 97.4%，在混淆 JavaScript 上为 87%；在 IOCCC 去混淆任务中，GPT-3.5 约 20%、GPT-4 19% 的目标得到可编译代码，最终仅分别有 8% 和 4% 产生正确结果。恶意软件分析是针对少量样本的定性案例研究：WannaCry 需人工分块输入，上下文被拆分时会漏掉行为，僵尸网络仓库中还出现误报。模型能认出来自热门仓库的代码，因此记忆效应是混杂因素。被评测的 GPT-3.5／GPT-4 时代模型及 LLaMA-2-13B 级别模型已经过时。研究分析的是源码或已反编译的文本，并不执行二进制反编译。", "targets": [ "website", "app" ], "title": "Large Language Models for Code Analysis: Do LLMs Really Do Their Job?", "title_zh": "面向代码分析的大语言模型：LLM 真的胜任吗？", "url": "https://www.usenix.org/conference/usenixsecurity24/presentation/fang", "venue": "USENIX Security 2024", "verification": "source-reviewed", "verified_at": "2026-10-18", "why_fits_en": "Measures how far LLM explanations survive JavaScript obfuscation and decompiler output, complementing JavaScript deobfuscation benchmarks and tools in the collection.", "why_fits_zh": "衡量 LLM 的代码解释在 JavaScript 混淆和反编译器输出下还能保持多少可靠性，与合集中的 JavaScript 去混淆基准和工具互补。", "year": 2024 }