Compares human and LLM-agent understanding and navigation on the same static reverse-engineering challenges, then studies how binary representations consume context windows.
Understanding is graded on two compact challenges; the separate 4,000-binary study measures context capacity, not task-solving accuracy. The interface is static IDA/MCP without dynamic execution, and prior exposure to challenge source remains an acknowledged validity threat.
Authors & citation
Samuele Doria; Stefano De Rosa; Ruoyu Wang; Zion Leonahenahe Basque; Eleonora Losiouk; Simone Aonzo
CTF-Dojo is an execution environment for training LLM agents on capture-the-flag tasks: 658 challenges from 50 competitions held between 2011 and 2025, taken from pwn.college's CTF Archive and containerized with Docker. Its CTF-Forge pipeline prompts DeepSeek-V3-0324 to write each challenge's Dockerfile, docker-compose file and challenge metadata; across three independent runs, 650 challenges consistently built and answered on their expected ports. Qwen3-Coder and DeepSeek-V3-0324 agents in the EnIGMA+ scaffold, capped at 40 turns, collected trajectories. Matched CTFtime writeups served as hints during collection and were removed afterwards, and ports, paths, distractor code and other runtime details were randomized per rollout. Rejection-sampling fine-tuning of Qwen3 8B, 14B and 32B on 486 flag-capturing trajectories from 274 solved challenges raises average Pass@1 on InterCode-CTF, NYU CTF Bench and Cybench by up to 11.6 points; the 32B model reaches 31.9%.
The environments span general CTF categories, not reverse engineering alone: of the 658 challenges, 228 are cryptography, 163 pwn, 123 reverse engineering, 85 miscellaneous, 38 forensics and 21 web. Among the 274 challenges with successful trajectories, cryptography is the largest category (141), followed by reverse engineering (52). Benchmark gains are reported per benchmark over 323 tasks, at temperature 0 with 40 turns, not per category, so they do not measure reverse-engineering accuracy. Decontamination matched competition and challenge names and removed 3 overlapping challenges; the authors note that it cannot exclude broader similarity between tasks. Only 150 challenges (23%) matched a writeup, all trajectories come from two teacher models, and only supervised fine-tuning is studied; the authors name the static 658-challenge scale as a limit. Hashed flag checkers in the upstream archive can be wrong, and the authors report four such bugs. The repository code is CC BY-NC 4.0. It holds CTF-Forge, the metadata and writeup-matching scripts and a challenge index, but not the challenge files, which are cloned separately from pwn.college's CTF Archive and are not covered by the repository license. The reviewed tree has no trajectories or trained weights, and writeups must be collected separately. Capturing a flag does not show recovered program structure.
Authors & citation
Terry Yue Zhuo; Dingmin Wang; Hantian Ding; Varun Kumar; Zijian Wang
Evaluates reverse-engineering agent traces through flag recovery, eight analysis milestones and action profiles. Deterministic checks and an outcome-blinded LLM judge help locate failures beyond a single solve-rate score.
Uses 88 public CTF challenges and nine models; all four prompting strategies are compared on four models. Evaluation fixes the D-CIPHER harness and uses one attempt per setting. One expert checks 15 traces; agreement on judged comprehension stages is 53/60. Recall and surface-perturbation probes do not eliminate contamination.
Authors & citation
Hadjer Benkraouda; Hongyu Cai; Berkay Celik; Gang Wang
Introduces SRE-Bench, with privately authored programs and protected binary variants, to study agent recovery of program semantics while reducing training-data contamination.
The revised study reports 19 programs, 262 binary instances, and 1,572 graded tasks. Binaries and task instances are not publicly released; access is through an evaluation service. Its contamination-control claim is the authors' methodology, not an independent guarantee.
Authors & citation
Jeremy Spence; Nicholas Assaderaghi; Feng Xiao; Jinhao Zhu; Nikil Ravi; Xiangyu Qi; Matthew Jagielski; Raluca Ada Popa; Eric Wallace; Guannan Wei; Yangruibo Ding; Zhuo Zhang
A proof of concept for indirect prompt injection against Cline/GhidraMCP binary-analysis agents. An adapted AutoDAN generates embedded strings that make the agent describe different behavior. The authors demonstrate four small C programs with Qwen3-8B and transfer the examples to GPT-OSS-120B.
Four illustrative programs, each with one or two functions; no repeated-trial success rate, realistic malware corpus or defense evaluation. The reported decompiler setting truncates strings beyond 2,048 characters. These examples do not establish general reliability across models or complex binaries.
Evaluates agents on executable-checked CrackMe tasks: infer validation logic from binaries and submit an accepted input or artifact, while recording tool use, time, and scoring attempts.
The v0 study uses 12 generated tasks and eight public calibration tasks, Linux x86-64 ELF binaries, a shell interface, and a five-minute budget. Its compact educational tasks do not establish performance on large applications. No paper-linked public artifact URL was verified.
2026Workshop on Binary Analysis Research (BAR) 2026Conference paper
Runs Claude Code, Codex CLI and ChatGPT-5.2 Pro on 24 crackme-style challenges selected from 15 international CTF events held in 2025, and attributes their failures to stages of an Observe–Comprehend–Plan loop and to three protection effects: concealment, complication and misdirection. From the failure logs, the authors derive four recurring weaknesses (training bias, over-trust in observations, context limitation and plan persistence). They rerun both CLI agents without a decompiler and compare outcomes with in-competition solve rates.
The study covers 24 locally solvable Linux x86-64 ELF crackmes and three commercial agents (Claude Opus 4.5 and GPT-5.1-Codex-Max in the CLI tools, and an undisclosed ChatGPT model), evaluated in December 2025 with network access; runs that consulted public write-ups were excluded. The binaries are small relative to real-world software, and malware analysis and vulnerability discovery are out of scope. Each configuration had at most three attempts with early stopping, which the authors call noisy; success means printing the correct flag or finding an accepted input. Weakness labels come from qualitative log analysis by two evaluators and describe these runs, not general properties of LLM agents. ChatGPT-5.2 Pro ran in an uncontrolled web sandbox and is excluded from the decompiler comparison. Solve rates are among teams that solved at least one challenge, unavailable for six tasks, and may include LLM-assisted teams. Training-data exposure remains a validity threat, and the paper does not publish its assembled challenge set, run logs or code.
Evaluates black-box program reconstruction across 200 tasks: agents query an executable and read its documentation, then independently implement a codebase whose behavior is checked by generated tests.
Binary inspection, decompilation, and internet access are forbidden during inference. Passing finite tests does not prove complete behavioral equivalence or recovery of the original source. Later live-leaderboard runs are separate from the reviewed arXiv results.
Studies how human analysts work with LLM assistance during static software reverse engineering. A survey of 153 practitioners informed six LLM features built into an instrumented IDA Pro plugin. Then 48 participants (24 self-reported novices and 24 experts) analyzed two CTF-style binaries, with the features enabled on one randomly assigned challenge. From graded writeups, interaction logs and surveys, the authors report 18 findings: novice understanding rose by about 98% to expert levels while experts gained little, known-algorithm functions were triaged up to 2.4× faster, and artifact recovery rose by at least 66%, alongside harmful hallucinations.
A human study of LLM-assisted analysis, not an evaluation of autonomous agents or of LLM decompilation accuracy. It uses two unobfuscated C challenges of 26 functions each, static analysis in IDA Pro only (no debuggers or symbolic execution), researcher-written prompts, and a choice of GPT-4o, GPT-4o-Mini, GPT-4-Turbo, Claude-3-5-Sonnet or Gemini-Pro. The 2024–2025 sessions retain 48 of 51 participants; expertise is self-reported, and understanding is a rubric score of writeups graded by two researchers. LLM-created artifacts were not correlated with better understanding, and three LLM users reported a nonexistent buffer overflow that an LLM had suggested; the vulnerability-finding feature was negatively correlated with understanding. Individual observations, writeups and survey responses are not released, even on request; aggregated data is available only on request.
Authors & citation
Zion Leonahenahe Basque; Samuele Doria; Ananta Soneji; Wil Gibbs; Adam Doupé; Yan Shoshitaishvili; Eleonora Losiouk; Ruoyu Wang; Simone Aonzo
Cyber-Zero synthesizes training trajectories for capture-the-flag (CTF) agents from public writeups instead of running the challenges. Writeups collected from CTFtime through the CTF Archives repository are cleaned, filtered and completed with DeepSeek-V3-0324, which fills in missing metadata and extracts flags; after removing 47 challenges that overlap the evaluation benchmarks, the corpus holds 6,188 writeups for 4,610 challenges from 543 competitions in six categories, 2017–2025. Two DeepSeek-V3-0324 personas then simulate each solve: a CTF player that sees only the challenge content, and a Bash terminal that sees the writeup and flag, invents plausible command output and may inject hints, which are removed before training. Trajectories use the EnIGMA scaffold format and are kept only if they reach the exact flag, pass format checks and are judged by an LLM filter to follow the writeup's approach. Qwen3, Qwen2.5-Instruct and SWE-agent-LM models are fine-tuned on 9,464 of these trajectories and evaluated with EnIGMA+, the authors' parallel version of EnIGMA, on InterCode-CTF, NYU CTF Bench and Cybench. The fine-tuned Qwen3-32B (Cyber-Zero-32B) rises from 20.3% to 33.4% average Pass@1, the 13.1-point gain reported in the abstract.
No training trajectory was executed: command outputs are simulated by an LLM that knows the writeup and flag, and trajectories are filtered by exact flag match, format checks and an LLM judge rather than by running commands. The authors acknowledge that the corpus may contain incorrect command traces and fabricated flags, that they report no quantitative error analysis, and that the simulated traces exclude interactive tools such as gdb, Python shells and browsers. The writeups and the evaluation tasks span six general CTF categories, not reverse engineering alone; the three test benchmarks hold 323 runnable tasks, 84 of them reverse engineering, and aggregate Pass@1 is not reverse-engineering accuracy. Results are single greedy Pass@1 runs in the authors' EnIGMA+ scaffold with a 40-turn cap, on benchmark copies the authors repaired; 17 tasks could not be set up. Contamination control removed 47 overlapping challenges by manual comparison. Capturing a flag does not show recovered program structure. The repository is archived and licensed CC BY-NC 4.0, and its benchmark README states that the bundled benchmark content remains the property of its original creators.
Authors & citation
Terry Yue Zhuo; Dingmin Wang; Hantian Ding; Varun Kumar; Zijian Wang
Benchmarks cryptographic binary reverse engineering across algorithm recognition, key or IV extraction, wrapper reconstruction, and flag recovery using 432 controlled challenges.
The corpus combines 48 algorithms, three insecure key-use scenarios, and three difficulty levels. It excludes professional-grade obfuscators such as Tigress and O-LLVM; results should not be generalized to heavily protected arbitrary binaries.
EnIGMA extends SWE-agent for capture-the-flag challenges with interactive agent tools that keep a gdb debugging session or a pwntools server connection open across turns, alongside decompile and disassemble commands, an LM summarizer for long outputs, and category-specific demonstrations and guidelines. It is evaluated with four LMs on 390 challenges from NYU CTF, InterCode-CTF, Cybench and a HackTheBox set the authors collected. The paper ablates tools, summarizers and demonstrations, and analyzes solution leakage and “soliloquizing”, where the model writes its own observations instead of acting in the environment.
The benchmarks mix cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks (104 of the 390 are reverse engineering), so aggregate solve rates such as 13.5% on NYU CTF are not reverse-engineering accuracy. The per-category table reports 17.65% on NYU CTF’s 51 reverse-engineering challenges with Claude 3.5 Sonnet. Results are pass@1 with a $3 per-instance cost cap and 2024 models (GPT-4 Turbo, GPT-4o, Claude 3.5 Sonnet and LLaMA 3.1 405B Instruct); the NYU CTF baseline comparison uses pass@5. Many challenges have public write-ups, and the authors’ leakage and soliloquy measures are heuristics; on 21 later CSAW 2024 challenges the best configurations solved 2. Solving a CTF shows a captured flag, not recovered program structure or source.
Authors & citation
Talor Abramovich; Meet Udeshi; Minghao Shao; Kilian Lieret; Haoran Xi; Kimberly Milner; Sofija Jancheska; John Yang; Carlos E Jimenez; Farshad Khorrami; Prashanth Krishnamurthy; Brendan Dolan-Gavitt; Muhammad Shafique; Karthik R Narasimhan; Ramesh Karri; Ofir Press
Cybench turns 40 professional-level capture-the-flag tasks from four competitions (HackTheBox Cyber Apocalypse 2024, SekaiCTF 2022–23, GlacierCTF and HKCert) into runnable agent tasks. Each task has a description, starter files, a Kali Linux container for the agent with task servers in Docker, and the time the first human team needed to solve it, from 2 minutes to 24 hours 54 minutes. Each task is also split into subtasks that grade intermediate steps. The authors evaluate eight models with a structured bash agent and compare four scaffolds for GPT-4o and Claude 3.5 Sonnet. Without subtask guidance, the best agents solved only tasks whose first human solve took at most 11 minutes, and Claude 3.5 Sonnet reached 17.5% unguided success.
Only 6 of the 40 tasks are reverse engineering; the rest are cryptography (16), web (8), forensics (4), miscellaneous (4) and pwn (2), so aggregate rates are not reverse-engineering accuracy, and Cybench is not a reverse-engineering-only benchmark. In the unguided structured-bash runs, six or seven of the eight models solved the two easiest reversing tasks (first human solves in 2 and 6 minutes), and none solved the other four. Results cover 2024 models (GPT-4o, o1-preview, Claude 3 Opus, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 and 3.1, Mixtral 8x22b) with the authors' scaffolds: 15 iterations unguided or 5 per subtask, short memory, no decompiler tools, one attempt in the main comparison and the best of three in the scaffold comparison. First-solve times come from competition records and are not directly comparable across competitions. The authors judge train–test overlap by model knowledge cutoffs and cannot rule it out for Claude 3.5 Sonnet. CTF tasks are short, small-codebase competition problems, and capturing a flag does not show recovered program structure.
Authors & citation
Andy K. Zhang; Neil Perry; Riya Dulepet; Joey Ji; Celeste Menders; Justin W. Lin; Eliot Jones; Gashon Hussein; Samantha Liu; Donovan Jasper; Pura Peetathawatchai; Ari Glenn; Vikram Sivashankar; Daniel Zamoshchin; Leo Glikbarg; Derek Askaryar; Mike Yang; Teddy Zhang; Rishi Alluri; Nathan Tran; Rinnara Sangpisit; Polycarpos Yiorkadjis; Kenny Osele; Gautham Raghupathi; Dan Boneh; Daniel E. Ho; Percy Liang
2024NeurIPS 2024 Datasets and BenchmarksConference paper
NYU CTF Bench packages 200 manually validated challenges from CSAW CTF qualifying and final rounds held between 2017 and 2023, spanning cryptography, forensics, pwn, reverse engineering, web and miscellaneous tasks, with challenge.json metadata and Docker images for server-side challenges. The paper also presents an automated framework that gives LLMs function-calling tools for shell commands, file creation, Ghidra disassembly and decompilation, flag checking and giving up. It evaluates five LLMs and compares their scores with human teams at CSAW 2022 and 2023.
Only 51 of the 200 challenges are reverse engineering (31 qualifying and 20 final), so overall solve rates are not reverse-engineering accuracy. All challenges come from one competition series, categories are imbalanced, and the remaining 367 of the initial 567 challenges, including incident-response tasks, are excluded pending validation. The baseline counts a challenge as solved if any of five attempts succeeds, within a 48-hour limit per model; no statistical tests were run. Among GPT-3.5, GPT-4, Claude 3, Mixtral and LLaMA, the best reverse-engineering rate was 9.80% (GPT-4), and the open models solved nothing. The human comparison uses competition scores from four CSAW events, and the public repository has continued to change after publication, so later runs may not match the paper’s challenge set or framework.
Locates patch-relevant code in stripped binaries using instruction-aligned retrieval and caller/callee context, then uses an LLM to screen candidate matches and compare patch-specific behavior with vulnerable and patched references.
Requires the known source patch and vulnerable/patched reference binaries; it is not source recovery or undisclosed-vulnerability discovery. Evaluation covers 62 CVEs from four C/C++ projects, 124 reference and 496 target binaries, restricted to x86-64, GCC/Clang configurations and one reported reasoning model. Balanced labels do not reproduce deployment prevalence; public patches may be in training data. Uncertain and missing-localization outcomes are reported separately, but binary metrics map them to vulnerable and negative outcomes, respectively; selected accuracy excludes both.
BScout matches decompiled functions to a source corpus using chunked embeddings and inlining-aware training. Existing matches provide positional clues to expand candidates; LLM selection and structural filters refine the resulting correspondences.
BinKit-BCSA covers 3,904 small-utility binaries; Large-10 covers larger projects only at x86-64/O2. Some anomaly thresholds come from FreeCAD in Large-10. Incomplete ground truth, sparse anchors and source-corpus coverage limit conclusions. Function chunks are encoded independently, and extra refinement trades precision and cost against recall. This is correspondence retrieval, not proof of source recovery or semantic equivalence.
Compares direct LLM recovery of Apple PBCodable schemas with an AI-written Extractor checked against compiled synthetic fixtures, published descriptors and captured traffic. It shows how plausible recovered schemas can silently decode values incorrectly.
The direct-agent study uses 17 seeds with 7,788 field occurrences, 4,987 distinct. Apple's true schemas remain unpublished; synthetic tests share the recovered template's assumptions. Traffic comes from one jailbroken iPhone 8 on iOS 16.7.11. Matching fields in newer OS binaries does not verify their network behavior. The maintained code also documents estimated or unresolved fields.
This SoK organizes LLM-assisted source-recovery designs and separates compilation, readability, structural and behavioral metrics. Its controlled study varies architecture, compiler optimization and symbol availability, with smaller ablations for input representation, model roles and source language.
The 45,000 cases are variants of 500 functions, not independent programs. Input/language ablations use 50 functions; translated language variants were produced with an LLM and checked using finite I/O tests. The study assumes a derivable call interface, is prompt-sensitive and cannot exclude ExeBench pretraining exposure. Test agreement and relocation-masked byte matching have different assumptions.
Authors & citation
Varun Kohli; Lee Bing Cheng; Nur Hazim Ghazali; Gao Yuze; Daryl Poon; Dinil Mon Divakaran
CodeInverter studies function-level LLM decompilation using control-flow graphs and memory-to-data mappings extracted with IDA/objdump. It presents a prompt workflow, training corpus and model family, evaluated through compilation, benchmark input/output tests and edit similarity.
Evaluation covers individual C functions under 32/64-bit GCC settings. Whole-program recovery, aggressive optimization and protected binaries remain limitations. The text reports one epoch (15 steps), while the pinned launch example sets one epoch and 100 warmup steps; the exact training schedule remains unresolved.
Authors & citation
Peipei Liu; Jian Sun; Rongkang Sun; Li Chen; Zhaoteng Yan; Xiaoling Zhang; Dawei Wang; Dapeng Sun; Peizheng Zhang; Dan Li
BinJudgeBench collects expert judgments for function naming, binary-code summaries and decompilation refinement. BinJudge learns to select a judge configuration for each input and task, evaluating agreement with human scores rather than executable correctness.
The router uses five-fold validation on 1,233 annotated outputs. Initial annotators are three authors, with an independent industrial expert checking a subset. Human scores are subjective, and fluent unsupported answers can mislead judges. Correlation on these tasks neither proves program equivalence nor establishes that judges can replace human review.
Authors & citation
Xiuwei Shang; Li Hu; Xiao Jiang; Jieke Shi; Junda He; Zhou Yang; Shaoyin Cheng; Guoqiang Chen; Weiming Zhang; David Lo
Quelle recovers library calls, loops, functions and symbolic parameters from compiled OpenQASM. It removes routing artifacts, proposes structured reconstructions and checks them against circuit semantics, retaining gate-level fallbacks when recovery fails.
Scope is quantum circuits with ideal-device semantics, not original-source identity. Dense checks use a 1e-6 tolerance; larger circuits additionally trust rewrite bookkeeping, an unchecked wide H-conjugation identity and QCEC. Dynamic circuits and timed-out checks remain unverified. Algorithm naming needs templates, and deliberate obfuscation is outside the evaluated scope.
ASAP aligns assembly and source representations during training, compresses assembly features into tokens, and supplies them alongside decompiler pseudocode to an LLM refiner. Masking and an assembly-advantage objective encourage the refiner to use low-level evidence.
Evaluation uses C adaptations of HumanEval and MBPP, x86-64 GCC 11.4 at O0–O3, and Ghidra 11.0.3 pseudocode. Re-execution means passing supplied tests, not proving equivalence. Identifier/comment recovery is not a dedicated component, and frontier-scale model training is not exhaustively evaluated.
Compares neural-decompiler C output with deterministic decompilation or reference functions using symbolic execution and a specialized memory model. It supports both analyst-facing validation and model evaluation without requiring the surrounding codebase.
Function-level checks abstract away side-effect ordering and some compiler details; configurable options further relax return, argument and type conditions. The implementation has explicit heap-related soundness limitations and requires compatible loop structure; unsupported C features and context-independent dependency assumptions also limit coverage. Failure to establish equivalence is not proof of inequivalence. Evaluations use filtered realtype and ExeBench pairs, not unrestricted whole-program equivalence.
Studies development of the Extractor reverse-engineering tool through commits, available session logs and code-provenance tracing. Taxonomies separate instruction intent, change triggers and agent claims, exposing errors corrected before commits that ordinary repository history misses.
One codebase with human-directed work, no human-coded comparison. The 14.3% figure is 15 of 105 audited generation events, including test defects and stale tests, not a defect rate per source line or in shipped code. LLM scoring has sampled human calibration; transcripts are incomplete and the archive omits per-response evidence bundles. This companion study does not independently corroborate the related schema-recovery results.
Echo iteratively generates source and compiler configurations, recompiles candidates, and uses assembly differences to guide configuration search and code refinement. It evaluates matching on function benchmarks and selected functions from a Mirai binary.
Matching means equality of normalized function assembly under evaluated compilation settings, not byte-for-byte recovery of a whole executable. Tests cover selected x86 GCC/LLVM settings and 62 Mirai functions; passing finite harness tests is not a general equivalence proof. Long optimized functions and configurations outside the searched space remain limitations.
Authors & citation
Jun Bi; Xiangxin Fang; Aarsh Chaube; José Wesley De Souza Magalhães; Rodrigo C. O. Rocha; Michael O'Boyle
Studies whether LLM refinement of decompiled C recovers identifiers from input evidence or prior knowledge. Within-item ablations and output-matched chance baselines test naming gains on textbook and newly authored functions, including a second-model replication.
The study uses 25 textbook and 20 author-written small functions, GCC/Ghidra on x86-64, and one prompt per model. Operations remain visible and the ablation also changes idiom-recognition cues, so its bounded null does not prove memorization or absence of input use. No registered hypothesis was confirmed; the correctness measure failed its discrimination gate; original T3 per-item artifacts were lost and re-executed, changing precision. Most findings use one refiner, and the later second-model extension is distinct from the initial preregistration.
Decompile-Diverge builds function-level behavioral comparisons from source-derived drivers and fuzzed reference inputs. The study compares reconstructed functions on the same inputs and examines divergences and missing crashes that compilation and supplied tests can miss.
The oracle requires reference source, supported inputs and buildable harnesses. It compares a finite corpus and bounded observable state; Matched does not prove all-input equivalence. Nine of 300 GitHub reference functions failed harness builds, leaving 291. Some types/functions are excluded. DeGPT-Qwen is a substituted-backend configuration, not the originally published DeGPT model.
ReSource refines decompiled functions affected by compiler optimization and obfuscation. Its stages retrieve guidance from a Semantic Distortion Database, predict a control-flow skeleton, and refine names and formatting. Evaluation measures code similarity and retrieval of the corresponding original function from a source corpus.
Function-level similarity and retrieval are proxies; they do not establish compilability, executable equivalence or complete program recovery. Section 5.2 notes missing global context and untested custom obfuscation. The author-linked artifact page describes a simplified dataset and explicitly says the database-construction and control-flow-prediction modules are not yet available.
STILL predicts function-level STL container presence from stripped control-flow graphs and turns those predictions into compact hints for LLM refinement of Ghidra output. StlBench supplies controlled labels for five container classes.
The interface covers map, queue, set, string and vector, not full variable types, template arguments, nesting, iterators or std::array. Experiments use g++/x86-64/libstdc++ with angr and Ghidra; transfer to other toolchains is unverified. Test-suite executability is not equivalence, and hint benefits depend on the downstream model and adaptation.
This survey maps traditional, dynamic, matching and neural decompilation approaches, then organizes type recovery, control-flow restructuring, naming and refinement tasks alongside evaluation metrics and benchmark choices. An accompanying repository provides the reviewed literature list.
The search ends in December 2025 and focuses on peer-reviewed studies of C-origin binaries. Selection and categorization involve manual judgment. Reported tool capabilities reflect surveyed publications, not necessarily current implementations. It is a literature synthesis, not a newly reproduced cross-tool performance benchmark or a complete account of 2026 preprints.
BRIDGE lifts stripped, optimized x86-64 and ARM64 functions, given as llvm-objdump disassembly, to LLVM IR. Its target is the unoptimized IR that a fixed clang -O0 configuration emits for the original source. Offline, it builds a retrieval database from 69,253 ExeBench functions compiled at -O0 to -O3, using LLVM pseudo-probe instrumentation to align assembly fragments with the matching -O0 IR regions. At inference it splits each function's angr-recovered control-flow graph into control-flow segments such as loops and if–else regions, retrieves the three nearest aligned assembly–IR pairs per segment from a FAISS index of Nova-1.3B embeddings, and prompts a hosted general-purpose LLM (DeepSeek-chat) to write the IR. Up to five refinement rounds then feed LLVM verifier and compiler diagnostics and unit-test failures back to the model. On HumanEval-Decompile and MBPP compiled with Clang 19 at -O0 to -O3, average re-executability is 65.2% and 65.4% on x86-64 and 45.6% and 42.1% on ARM64. The rule-based MCTOLL reaches 62.1% and 64.9% on x86-64, and the five directly prompted API LLMs reach at most 21.5%.
Evaluation is function-level on short benchmark programs: HumanEval-Decompile and an MBPP subset, with inputs of 2 to 628 assembly instructions, built by the authors with one LLVM/Clang 19 toolchain on Linux. Canonical means matching one fixed clang -O0 configuration, not recovering the original source. Refinement and the re-executability metric both use the unit tests these benchmarks supply; the authors note that such test oracles are rarely available in practical reverse engineering. On x86-64 HumanEval-Decompile, re-executability falls from 65.2% to 60.8% with LLM-generated tests and to 53.5% with no tests. Output still diverges structurally from the reference: average edit similarity stays below 50%, and exact control-flow-graph matches reach 25.9% on that benchmark. The pipeline calls a hosted LLM API, as do the LLM baselines; results are geometric means of five runs and can shift as hosted models change. Forklift and LLM Compiler FTD are excluded from the baselines because the authors treat them as lifting compiler-emitted assembly text, and MCTOLL is compared only on x86-64. Other instruction sets need a rebuilt retrieval database. BRIDGE produces LLVM IR, not C source, so it is not a decompiler to C. The code repository has no license.
Authors & citation
Xiaoao Zhu; Jie Ren; Zhiqiang Li; Jie Zheng; Zhanyong Tang; Zheng Wang
2026Proceedings of the ACM on Software Engineering (FSE 2026)Journal article
The paper builds asmFooler to test whether learned binary code similarity detection (BCSD) models keep matching functions after semantics-preserving changes. From 620 baseline x86-64 ELF binaries (coreutils, binutils, PuTTY, nginx, SPEC2006 and others, built with Clang at O0–O3), it generates 9,565 variants using in-place instruction rewriting, basic-block reordering, Obfuscator-LLVM, semantic NOPs and junk code, and it adds a greedy search for dead-code sequences that make dissimilar functions look similar. Across Genius, Gemini, Asm2Vec, SAFE, Trex and BinShot, robustness depends on each model's preprocessing, architecture and features: block reordering drops BinShot's recall from 0.972 to 0.036 but lowers it by under 0.05 for the CFG-based Genius and Gemini. The false-positive attack reaches up to 100% success with 14.75 inserted instructions on average.
All samples are x86-64 ELF binaries compiled by the authors from open-source packages and SPEC2006; some packages failed to compile or transform, and the in-place transformation is non-deterministic and not formally verified. The six models evaluated (Genius, Gemini, Asm2Vec, SAFE, Trex and BinShot) exclude jTrans and LLM-based similarity models, so the findings are not shown to transfer to newer or LLM-based systems. Results depend on the transformation budgets the authors set to fit each model's input limits, and semantic NOPs and junk code are inserted only at function entry. Packing, encryption, string obfuscation and commercial protectors such as VMProtect and Themida are out of scope, and adversarial training is left to future work.
Tests reimplementation of 25 command-line programs using execute-only reference access, documentation and visible examples. Separate scoring environments compare exact outputs on visible and held-out end-to-end tests.
Targets are scoped, deterministic, stateless programs with text output. Test agreement is not universal equivalence or code-quality assessment. A memorization screen flags possible exposure for 17 targets. Large inference budgets and scaffold choices limit comparisons with cheaper benchmarks; three targets remain private.
Authors & citation
Tom Adamczewski; David Owen; David Rein; Florian Brand; Giles Edkins; Allen Hart; Daniel O'Connell
A systematization-of-knowledge survey of machine-learning, LLM and agentic approaches to binary reversing. Version 2 collects 246 papers published since 2015, found through keyword searches of 11 security, software-engineering and AI venues plus citation snowballing, and organizes them into 22 domains by inference task: from code-boundary detection and recovery of function names, types and struct layouts to code similarity, malware and vulnerability analysis, and decompilation. Its taxonomy links conventional triage, static, dynamic and security-testing analyses to AI pipelines through the binary-derived artifacts they exchange, and it derives 12 insights and open directions such as binary-aligned ground truth and goal-driven agentic reversing.
A literature systematization with no new experiments, benchmark or tool; it classifies and summarizes reported results without reproducing them. The corpus covers papers available up to July 31, 2026, applies citation thresholds that fall from 12 for 2015–2016 papers to 2 for 2025–2026 papers, excludes methods that need source code at inference time, and leaves out Android and interpreted-language bytecode. Detailed per-stage tables draw on 88 top-venue studies rather than the full corpus. It is an arXiv preprint with no stated venue.
AutoDecompiler trains function-level decompilation models to revise assembly- or pseudocode-derived outputs using compilation, execution and I/O feedback. Its reinforcement-learning design combines diagnostic feedback with rewards for valid code, tested behavior, syntax and data-flow structure.
Evaluation uses C adaptations of HumanEval and MBPP with validation feedback; finite tests do not establish general equivalence. HumanEval also informs reward-weight selection, and base-model pretraining exposure cannot be ruled out. Refinement is context-limited, whole-program dependencies are not recovered, and unseen toolchains or protected binaries remain limitations.
Authors & citation
Peipei Liu; Jian Sun; Mingzhe Xing; Yicheng Zeng; Zhaoteng Yan; Lixiao Zhang; Li Chen; Dan Li
Uses a composite Quantitative Readability Score (QRS) to guide agents refining decompiled C. Lexical familiarity, structural simplicity and coding idioms are combined with a binary-structure similarity gate. Experiments compare refinement with and without shell access on 210 simple C binaries.
Structural similarity does not prove semantic equivalence. QRS both guides and evaluates refinement; human readability validation remains future work. The selected synthetic corpus limits generalization. Recovering compilable, behaviorally correct code from difficult binaries is outside the claimed contribution.
Benchmarks LLM-based EVM decompilation on 600 source-verified Ethereum Solidity contracts, sampled in easy, medium and hard thirds from 772,736 unique bytecode–source pairs after near-duplicate filtering. Each task supplies EVM assembly and optional four-byte selector hints and asks for self-contained Solidity with compiler settings. Outputs are scored in four cumulative stages: format completeness, compilation, ABI recovery, and semantic consistency, where 227,383 recorded test cases for 14,553 public functions are replayed on the recompiled output. With one compiler-feedback repair call, GPT-5.3-Codex compiled 90.3% of contracts, while the best semantic results were 29.4% of functions and 42 of 600 contracts recovered perfectly, both by Claude Opus 4.7.
Replay compares return data, reverts, logs and touched storage on generated inputs that reach 77.8% average code coverage; passing it is not proof of equivalence, and replay needs an Ethereum archive node. The contracts are source-verified Ethereum Solidity from compiler v0.4 to v0.8, so unverified, obfuscated, Vyper or other-chain bytecode is not covered. Three models are evaluated zero-shot through OpenRouter: Claude Opus 4.7 and GPT-5.3-Codex at high reasoning effort, and GLM-5 with and without extended thinking. Results and quoted costs describe those 2026 snapshots. Gigahorse and Heimdall-rs are scored only on selector and prototype recovery because they do not emit compilable Solidity; DiSCo and the fine-tuned decompiler of David et al. are discussed but not run.
Uses unpacking outcomes as reference evidence to assess packer labels, normalizes heterogeneous tool results, and evaluates complementary signature/heuristic pools with static predicates derived from unpacker implementations. It traces how identification errors affect later unpacking and malware classification.
The corpus is approximately 160,000 Windows PE samples, split into roughly 130,000 for analysis/rule construction and 30,000 for testing. Unpacker failures or partial recovery can compromise the derived reference labels. Family-identification improvements use common pooled-rule configurations, not independently validated patches to seven separate tools. Repair evaluation excludes proprietary implementations and graph/learning-based methods. Results do not establish coverage of other executable formats, all packer families or general semantic equivalence.
Decaf samples multiple candidate decompilations, recompiles them and ranks them using compiler-derived byte distance or a learned reranker. The work studies candidate selection as a way to improve neural decompilation without an iterative source-repair conversation.
Byte matching assumes access to the reference compiler/configuration. Experiments subsample ExeBench and exclude examples Hex-Rays cannot decompile; stripped Synth is not execution-scored because call targets cannot be linked. Supplied tests can miss errors. Alternate architectures and obfuscated code are not evaluated, and Juliet results need not generalize to complex vulnerabilities.
Authors & citation
Alexander Shypula; Osbert Bastani; Edward Schwartz
NotDec lifts WebAssembly into SSA-based IR, uses Retypd-derived type constraints and pointer/value distinctions to recover structures, and emits C. Its study separates full interprocedural analysis from a faster intraprocedural variant.
The study uses Juliet and five Howard programs at O0/O3. Howard binaries omit actual library linking and cannot be executed, so recompilation does not demonstrate behavioral equivalence. Full analysis timed out on six Howard variants at 24 hours; recompilation results cover successful decompilations. The inspected README also warns about backend syntax errors on complex statements.
Authors & citation
Jikai Wang; Ningyu He; Tianming Liu; Junhai Wang; Haoyu Wang
HELIOS gives an LLM a hierarchical summary of binary control flow and function calls alongside decompiler output, with optional compiler feedback. It evaluates compilability and functional correctness.
Evaluates how language models interpret raw EVM bytecode from purpose-built deceptive and control contracts. It distinguishes recognizing a hidden transfer mechanism from correctly judging its risk, using model judges alongside judge-independent checks.
The main evaluation covers 22 models and 13 synthetic contracts using single-shot raw bytecode, without execution tools, source retrieval or on-chain context. Its 8,528 analyzed responses exclude refusals and pending runs. Gold labels come from one expert; some judges also belong to evaluated model families. Ground-truth checks vary across contracts, including owner/selector-only checks and no dedicated EVM tests for two injection variants. These results neither measure deployed-contract prevalence nor generalize to interactive reverse-engineering agents.
Splits LLM decompilation into two phases: a Structure Recovery model turns decompiler pseudocode into an identifier-free IR that keeps control flow and data structures, trained with compiler-based RL rewards, and an Identifier Naming model then restores names under a semantic-similarity reward. The authors report a 21.6% average re-executability gain over GPT-5-mini on HumanEval and a 29.4% average R2I improvement over Idioms on GitHub2025.
Training and main evaluation use function-level C compiled for x86-64 Linux with GCC and Clang at -O0 to -O3, with IDA Pro pseudocode as model input. Re-executability is measured only on HumanEval and MBPP; ExeBench and GitHub2025 use R2I readability and a GPT-5-mini judge. Appendix experiments extend to BringUpBench projects, macOS arm64 and Windows MSVC builds, and Obfuscator-LLVM, but the models are trained on C only and the paper expects them to degrade on features without C equivalents, such as C++ templates.
ShieldedCode studies software protection from the defender's side: it trains code language models on pairs of C source functions and their virtualization-protected (VMP) x86-64 code so that the models can generate, compare and rank protected variants. Training pairs come from AnghaBench functions compiled at O0–O3, protected at levels L0–L3 by a commercial VMP tool, then disassembled and normalized. The model adds attention masks for dependencies within, between adjacent and across virtual instructions, contrastive objectives for functional equivalence and protection strength, and a protection-effectiveness task that ranks variants of the same source. Continual pre-training starts from CodeLlama 34B and adds The Stack to the corpus, and fine-tuning uses 2.5 million source–VM pairs. On HumanEval-compile, VM code generation reaches 26.95% Pass@1 at level L0 (GPT-4o: 22.58%) and 14.71% at L3. On BinaryCorp-VirtualAssembly similarity detection, Recall@1 for the O0+L1 setting is 0.488, against 0.333 for jTrans with a linear probe. The authors also run a manual reverse-engineering study and pattern-matching and angr attacks on the generated protection.
All data is x86-64, and every training pair is produced with one commercial VMP tool that the paper does not name, so results may not carry over to other architectures, protectors or virtual-instruction designs. Generated VM code is checked with HumanEval-compile test cases, not formal equivalence, and Pass@1 stays between 14.71% and 26.95% across protection levels. The similarity gain over jTrans holds only in the O0+L1 setting: in the other five BinaryCorp-VirtualAssembly settings the linear-probe jTrans has higher Recall@1 (for example 0.715 against 0.309 for O2+L1), and the paper's summaries of the comparison differ (a 10% gain in the abstract, 15.5 points in the introduction, competitive in the results). The ablation table reports scores for ContraBin, CLAP and CEBin that the paper says are estimated from their architectures rather than measured. Protection-strength claims rest on the authors' own attacks and manual study: 12 graduate students and 3 professional reverse engineers analyzing 20 functions for 30–45 minutes each, with 17% success on ShieldedCode output against 67% for VMProtect and 58% for Themida. ShieldedCode is not a devirtualizer or deobfuscator, and the paper links no code, models or generated training data.
Authors & citation
Mingqiao Mo; Yunlong Tan; Hao Zhang; Heng Zhang; Yangfan He
Idioms fine-tunes code language models to turn Hex-Rays output for one target function into predicted original C together with definitions of the user-defined structs, unions and typedefs it uses. Input can include neighboring decompiled functions in breadth-first call-graph order, so evidence about a type scattered across callers and callees reaches the model. The authors also build Realtype, 157,163 functions from compiled open-source C repositories with full type definitions. Their CodeGemma-7b-based model passes ExeBench unit tests for 54.4% of functions versus 46.3% for LLM4Decompile and 37.5% for Nova, and scores at least 95% higher on Realtype.
This is local, function-level neural decompilation: each prediction covers one target function and its type definitions, not a compilable whole program. Evaluation uses x86-64 Linux gcc binaries from open-source GitHub projects and ExeBench’s “real” subset; O1–O3 builds disable inlining, and keeping only functions that decompiled at every level removes 21% of the Hex-Rays and 76% of the Ghidra test sets. ExeBench unit tests are an optimistic proxy; Realtype has no tests, so dependency-graph equivalence and exact name and type matches serve as conservative proxies, and a Realtype user-defined type counts as correct only if all of its fields, 17.6 on average when counted recursively, match. The main models use proprietary Hex-Rays output and context windows of 2,048 or 4,096 tokens. Predicted names and types are statistical, not recovered originals. Pretraining leakage from GitHub remains possible, and obfuscated code and malware were not evaluated.
FidelityGPT post-processes existing decompiler output: an LLM labels lines with six distortion types, such as non-conventional dereferencing, redundant variables and confusing control flow, and then rewrites the labeled lines. Its prompts define the distortion types and add retrieval-augmented context from a manually annotated database of distorted IDA Pro and Ghidra lines. A semantic-intensity algorithm chooses which lines to query, and a variable-dependency analysis carries redundant-variable context across 50-line chunks. On 620 function pairs, the authors report 89% detection accuracy and 83% precision, and correction fix rates of 94% (FR) and 64% (CFR) against 83% and 37% for their DeGPT runs.
The 620 pairs are mostly -O0 IDA Pro 7.5 functions sampled from Coreutils, curl, ImageMagick, OpenSSL, PuTTY and SQLite builds in a binary-similarity dataset, across ARM-32, MIPS-32 and x86-32, plus x86-64 TheAlgorithms/C code; separate -O1–-O3 and Ghidra sets test generalization. Ground-truth labels and FR/CFR were assessed manually by the author team against the paper’s criteria. Fix rates count lines marked as fixed and are not semantic-equivalence proofs: the authors note that repaired code often cannot be recompiled or executed. Correction runs on whole functions and may degrade on long ones, and the distortion database, built from 150 IDA Pro and 91 Ghidra lines, may miss rarer anomalies. Main results use GPT-4o at temperature 0.5 and remain stochastic; DeepSeek-chat and temperature-0 runs are reported separately. Training-data exposure of the open-source projects cannot be ruled out.
Authors & citation
Zhiping Zhou; Xiaohong Li; Ruitao Feng; Yao Zhang; Yuekang Li; Wenbu Feng; Yunqian Wang; Yuqing Li
CHISEL refines Ghidra pseudocode using compiler diagnostics and type-aware differential fuzzing against the original function. It retains counterexamples and candidate history; existing I/O pairs are held out for evaluation rather than supplied to the recovery loop.
Evaluation covers 120 selected ExeBench functions, x86-64 O0–O3, stripped/unstripped variants and a fixed Gemma4:31b model. The feedback oracle still falsely accepts candidates; finite fuzzing and held-out tests do not establish equivalence. Signature recovery and possible benchmark exposure remain limitations.
Authors & citation
Varun Kohli; N Raghava; Biplab Sikdar; Dinil Mon Divakaran
20252025 IEEE Annual Computer Security Applications Conference (ACSAC)Conference paper
The paper fine-tunes eight local models from CodeLlama, Qwen2.5-Coder, and CodeGemma on 5,981 x86 assembly examples and evaluates their help with binary reverse engineering.
Measures how well existing WebAssembly decompilers recover C, with native-binary decompilers as the baseline. The same C programs are compiled with Emscripten to WebAssembly and with clang to native code, then decompiled by wasm2c, w2c2 and wasm-decompile or by Ghidra and RetDec. Correctness reuses 1,000 Csmith programs from DecFuzzer, recompiling the output and comparing its execution with the original; code-complexity metrics and AST similarity to the original source are measured on PolyBenchC and CHStone at -O0 to -O2. wasm2c passed all 1,000 correctness programs, against 649 for wasm-decompile, 818 for Ghidra and 775 for RetDec.
Correctness covers only generated single functions with local-variable arithmetic and no library calls, pointers or global state, so the wasm2c result does not establish correctness on general programs. w2c2 is excluded from the correctness test, and wasm-decompile output could be parsed for structural similarity only at -O0. Native baselines pass through DecFuzzer’s syntax rewriter, and the authors note that some Ghidra and RetDec failures may come from that evaluation step rather than the decompilers. Readability is estimated from metrics such as lines of code, nesting depth, cyclomatic and Halstead complexity, gotos and casts, not from a human study. The benchmarks are C suites compiled by the authors with one Emscripten/clang toolchain, not real-world WebAssembly modules, and tool versions are described only as the latest available at the time.
Authors & citation
Wei-Cheng Wu; Yutian Yan; Hallgrimur David Egilsson; David Park; Steven Chan; Christophe Hauser; Weihang Wang
BinMetric poses 1,000 questions across six binary-analysis tasks: call-site reconstruction, decompilation, signature recovery, binary code summarization, algorithm classification and assembly instruction generation. The questions come from 20 open-source C projects compiled for x86-64, stripped and processed with IDA Pro, and aligned with source code through DWARF information. The authors evaluate 12 LLMs from five model families with one-shot prompts, alongside IDA Pro’s decompiler, LLM4Decompile, BinT5 and HexT5 where those apply.
Scores are question-level metrics on isolated snippets, not end-to-end reverse-engineering workflows or agentic tool use. Preprocessing requires IDA Pro, and all binaries are x86-64 builds with each project’s default compiler settings on Ubuntu 22.04; other architectures, optimization sweeps and obfuscation are not studied. Most tasks score text overlap with Rouge-L, BLEU, METEOR or CodeBLEU, so they do not test semantic equivalence; only assembly generation checks compilation and test-case execution, where the average execution pass rate is below 1%. Summary labels were generated by ChatGPT and then manually reviewed, and algorithm classification uses 80 snippets from a single algorithms project. Leakage screening relied on exact-match web searches. The evaluated models predate 2025 (including gpt-3.5-turbo-16k-0613 and gpt-4-0613) and ran with a maximum length of 8,192 tokens, so the rankings are historical.
Studies LLMs as source-to-source deobfuscators for C functions protected with the Tigress obfuscator. Functions from ExeBench, with identifiers randomized, are obfuscated with five transformations: MBA arithmetic encoding, branch encoding, control-flow flattening, opaque predicates and argument randomization. Each model receives a whole obfuscated function and returns simplified C. DeepSeek Coder 6.7B and Code Llama 7B are fine-tuned on obfuscated–original pairs, GPT-4 is prompted, and Clang optimization at -O0 to -O3 serves as a compiler baseline. Simplification is scored with Halstead program length relative to the original and obfuscated code, syntactic correctness by compiling ExeBench wrappers and semantic correctness by ExeBench input/output samples. The fine-tuned models reduce complexity more than GPT-4 on every transformation and more than Clang except on arithmetic encoding and argument randomization, where Clang is on par. Trained on chains of up to five transformations, they keep stable scores on chains of six and seven, and the best model cuts Halstead program length by 89.21% on average in the hardest scenario. A test with randomized constants found one sample that reproduced memorized code.
The obfuscation is Tigress (version 3.3.3 in the released scripts) applied source-to-source to single C functions. Compiled binaries, decompiler output, commercial protectors, inter-procedural obfuscation, VM-based packing and self-modifying code are not evaluated; the authors only note that decompilation can approximate C from binaries. This is source-level simplification, not binary deobfuscation or VM devirtualization. Semantic correctness is the main weakness: for single transformations the fine-tuned models preserve behavior in 50% to 94.5% of samples, mostly below GPT-4 and, except for branch encoding, below Clang, and it falls as transformation chains grow longer. Complexity scores count only samples that are both syntactically and semantically correct. Behavior is checked with ExeBench input/output samples rather than equivalence proofs, inputs are limited to a 6,144-token context, and models were fine-tuned with 8-bit quantization. Only one obfuscator is used, so generalization to other obfuscators, versions, machine code or bytecode is left open, and baselines are GPT-4 and Clang rather than dedicated deobfuscation tools.
Authors & citation
David Beste; Grégoire Menguy; Hossein Hajipour; Mario Fritz; Antonio Emanuele Cinà; Sébastien Bardin; Thorsten Holz; Thorsten Eisenhofer; Lea Schönherr
Evaluates traditional and LLM-based decompilers using functions from real programs, runtime behavioral checks, and an LLM-assisted assessment of analyst-facing code understandability.
The evaluated corpus has 23,400 functions from 130 programs. Fuzzing-based checks have finite coverage, and understandability scores depend partly on an LLM judge. The artifact warns that rebuilt OSS-Fuzz projects can differ from the paper's versions.
Converts Ethereum Virtual Machine bytecode into Solidity in two stages. Static analysis first lifts the bytecode into three-address code with recovered control flow; a Llama-3.2-3B model fine-tuned with LoRA on 238,446 TAC-to-Solidity function pairs then writes each function. On 9,731 held-out functions from verified Ethereum contracts, the authors report average semantic similarity of 0.82 to the original source, with 78.3% of functions above 0.8. An ablation compares the untuned base model on 663 functions, and case studies decompile an unverified Dx Protocol contract and an exploited MEV bot.
The 0.82 figure is the paper’s semantic-similarity score, whose computation v1 does not specify. With normalized edit distance and token-frequency comparisons, it measures resemblance to verified source, not behavioral equivalence; v1 reports no compilation, execution or differential tests of the generated Solidity. Training and test functions come from verified contracts, so closed-source deployments appear only in case studies. Comparisons with traditional decompilers are stated in general terms; no head-to-head experiment against a named decompiler such as Elipmoc is reported. The authors show a DeFi staking function whose output lost its fixed-point arithmetic and time-based reward logic (similarity 0.52), and report weaker results on very long functions and on contracts with deep inheritance or nested control flow.
Authors & citation
Isaac David; Liyi Zhou; Dawn Song; Arthur Gervais; Kaihua Qin
2025Proceedings of the ACM on Software Engineering (ISSTA 2025)Journal article
DecLLM repairs decompiler pseudocode that does not recompile into C that compiles and runs, so source-level tools such as CodeQL can be applied to binaries. An iterative loop feeds compiler errors to an off-the-shelf LLM, reverting illegal outputs and large deletions, then uses mismatched test outputs and AddressSanitizer reports from the rebuilt program to repair behavior, and finally fuzzes the original and rebuilt executables for divergences. With GPT-3.5 and GPT-4 the authors report 74% and 78.3% success on 300 Code Contest programs; on Coreutils and CGC binaries, 61.9–70.9% recompile and 48.1–59.0% are fully functional.
On Code Contest, success means the repaired code compiles and the rebuilt executable passes the supplied test cases, with AFL++ fuzzing as an extra check; the Coreutils and CGC overall rates also count rebuilt binaries that malfunction or crash. This establishes compilability and test-based agreement, not semantic equivalence, which the authors note is undecidable. Results depend on IDA Pro 8.3 output, with one Ghidra 10.4 repetition, and on the API-served gpt-3.5-turbo-0613 and gpt-4-0314 snapshots with a 4,096-token limit, so programs were chosen to fit within half of it. The 300 Code Contest programs average 112.9 lines; Coreutils 9.5 and CGC binaries were compiled without stripping symbols and repaired function by function, which cannot infer undefined globals. Around 30% of hard cases still fail, mainly from decompiler address/data confusion, ill-inferred buffer sizes, long outputs that include STL code and LLM type-cast shortcuts; deviations not exercised by tests remain undetected. CodeQL analysis of Juliet samples needed manual query adaptation.
Authors & citation
Wai Kin Wong; Daoyuan Wu; Huaijin Wang; Zongjie Li; Zhibo Liu; Shuai Wang; Qiyi Tang; Sen Nie; Shi Wu
2025Proceedings of the ACM on Software Engineering (FSE 2025)Journal article
DiSCo lifts Ethereum contract creation bytecode to Solidity source by prompting a frozen LLM, instead of emitting decompiler pseudocode. It abstracts control-flow paths into logic-invariant semantic units, recovers constructor logic by simulating creation bytecode captured from block execution, and infers stripped names with signature databases and a type-aware graph model. These units become natural-language descriptions plus coding specifications in the prompt. GPT-4 is the primary model; Llama 3 is also tested.
Ground truth comes from Etherscan source files: 200 highest-balance contracts for all baselines and 1,000 for automated tests; separate studies add 50 Vyper, 50 newly deployed and 149 closed-source attack contracts. Compilability (about 75% with best-of-five sampling, 46% for a single run), differential-fuzzing pass rates and embedding similarity do not establish equivalence to deployed source. Nearly half of the imitation-learning fuzzer’s transactions revert, and manual review judged about 28.5% of generated contracts semantically consistent, and 12% for Vyper inputs. The authors report timeouts or unsound analysis on contracts with many paths, and name inference remains weak. Baselines give the same LLM output from Panoramix, Ethervm and Elipmoc, mostly taken from their websites or decompiled manually; this is not a comparison of standalone decompilers. Results depend on a GPT-4 preview snapshot. Attack reproduction covers ten January 2025 incidents, four of which were reproduced.
D-LIFT uses D-Score to guide LLM fine-tuning and output selection, improving decompiled code while prioritizing accuracy. It evaluates Ghidra output from coreutils and util-linux.
The evaluation covers functions from coreutils and util-linux, using Ghidra and a range of LLMs. D-Score measured 55.3% more improved functions than baseline LLMs and improvements to 68.2% of native decompiler functions.
ReF Decompile relabels jump targets and uses function calls to recover variable information when converting binary code to source code. It was evaluated on the HumanEval-Decompile Benchmark.
The evaluation covers binary code decompilation on the HumanEval-Decompile Benchmark, with comparisons to similar methods and a reported performance of 61.43%.
Authors & citation
Yunlong Feng; Bohan Li; Xiaoming Shi; Qingfu Zhu; Wanxiang Che
GenNm suggests names for placeholder variables in decompiled code of stripped binaries by fine-tuning generative code models (CodeGemma-2B by default; CodeLlama-7B and a LoRA-tuned CodeLlama-34B are also tested). Queries add renamed caller code and callee signatures propagated along the call graph; a DPO-style symbol-preference optimization (SymPO) counters bias toward frequent training names, and a data-flow name-validation step chooses among candidates over iterative rounds. The authors report 5.6–11.4-point precision gains over VarBERT and ReSym on rebuilt DIRTY and VarCorpus datasets, and 22.8% precision on names unseen in training against 17.3% for the strongest baseline.
Predictions are scored against debug-symbol names by token matching with SymLM's word clusters and rules, plus a GPT-4 evaluator for context relevance and semantic accuracy; neither shows that outputs are the original identifiers, and overall split-by-binary precision is 35.8% (DIRTY) and 41.2% (VarCorpus). Precision is about 30% for projects unseen in training, and gains are smaller at -O2 and -O3. Ground truth comes from debug information in Linux binaries rebuilt from GitHub projects or Gentoo packages, with IDA Pro output in the paper's examples. GenNm needs each binary's call graph, which is why the DIRTY dataset was rebuilt; split-by-function VarCorpus results use a 15% binary sample, and the black-box GPT-3.5/GPT-4 comparison uses 1,000 sampled functions. It recovers variable names only, not types, function names or full decompiled code.
TypeForge recovers composite data types, including struct layouts, unions and pointer or nesting relationships, from stripped binaries. A whole-program Type Flow Graph over Ghidra’s decompiler-inferred variables propagates member offsets and sizes, cuts conflicting edges caused by casts and unions, and synthesizes candidate type declarations. An LLM only selects among these statically synthesized candidates: each is applied through the decompiler’s retype interface, and a double-elimination tournament compares the resulting pseudocode for readability. On OSPREY’s coreutils dataset the authors report F1 of 81.7% for composite-type identification, 88.2% for layout recovery and 72.1% for relationship recovery.
Recovers member offsets, sizes and relationships, not original type or member names; relationship correctness is judged by matching layouts because names are unavailable. The main evaluation uses OSPREY’s dataset of GNU coreutils and larger Howard-dataset programs, over 10,000 functions and 11,000 composite-typed variables at O0–O3, with ground truth from debug information; the paper does not report results per architecture. In the baseline comparison, OSPREY results were supplied by its authors and its runtime is an estimate, and ReSym figures are its published results, so not every baseline was rerun. Conservative removal of conflicting edges can drop valid members, and compiler optimizations that merge or flatten members lower accuracy at higher levels. About 75% of runtime is the LLM selection stage; the default model was GPT-4o mini, with each comparison run three times. Downstream readability and vulnerability examples are case studies, not proof of semantic equivalence.
Nova is a generative LLM for x86-64 assembly, trained from DeepSeek-Coder checkpoints on stripped, normalized gcc -O0 to -O3 assembly built from AnghaBench and The Stack. It adds hierarchical attention that builds per-instruction summaries, and contrastive objectives that keep a function's source and O0–O3 assembly close while ordering them by optimization level. Fine-tuned for decompilation, Nova-1.3B and Nova-6.7B reach 25.17% and 34.36% average Pass@1 on HumanEval-Decompile, 14.84 and 12.41 points above same-size LLM4Decompile; separately fine-tuned models report the highest average Recall@1 for O0-to-O3 binary code similarity on six real-world projects.
Decompilation is evaluated function by function on the 164 HumanEval-Decompile C problems, each compiled for x86-64 at O0–O3, by running 20 samples per function against the problems' test cases; Pass@k on these tests is neither whole-program nor equivalence evidence. Baselines other than LLM4Decompile and Nova were prompted with three-shot examples rather than fine-tuned. Similarity results cover O0 queries against O3 candidates in pools of 50–500 functions from Binutils, Curl, ImageMagick, SQLite, OpenSSL and PuTTY, after fine-tuning on BinaryCorp-3M. Decompilation fine-tuning used a 2.16M-pair sample, the README says only 50K files came from The Stack, and the paper limits Nova to x86-64 assembly and 1.3B and 6.7B models. Inputs require the paper's assembly normalization.
Authors & citation
Nan Jiang; Chengxiao Wang; Kevin Liu; Xiangzhe Xu; Lin Tan; Xiangyu Zhang; Petr Babkin
2025IEEE Transactions on Information Forensics and SecurityJournal article
MALSIGHT writes natural-language summaries of stripped binary malware one function at a time. IDA Pro produces pseudocode and a call graph, and functions are summarized callees first, so each caller receives its callees' summaries as context; a sequence-labeling model also marks key API calls and strings, which drive code-search retrieval and LLM-written annotations. The summarizer, MalT5, is a 0.77B-parameter CodeT5+ model fine-tuned first on MalS, 89,609 malicious C source functions from GitHub with LLM-generated summaries refined by human reviewers, and then on annotated benign pseudocode from the Capybara dataset. The authors also introduce MalP, 500 pseudocode functions for testing, and BLEURT-sum, a fine-tuned BLEURT metric, and report that MalT5 performs comparably to much larger Code Llama models.
Training summaries are LLM-generated and then reviewed and refined by five experts rather than written from scratch. MalP's 500 test functions come from compiling and decompiling 20 of the same GitHub malware repositories, not from wild samples, and the authors note that loose filtering may let training and test data overlap and inflate results. Quality is scored mainly with the authors' own BLEURT-sum metric. On ten real malware samples, with 79 functions also rated by three reverse engineers, BLEURT-sum was about 30% lower than on MalP, which the authors attribute to long functions: 15.8% exceed 1,000 tokens, while most training functions are under 300. The pipeline depends on IDA Pro pseudocode, and static annotation relies on online code search and a general-purpose LLM. MALSIGHT summarizes behavior; it is not a malware detector or classifier.
Lares tests whether a known 1-day vulnerability fix is present in a target binary function without recompiling the project. Its inputs are the target binary, the project source and the patch commit diff. A patch-enhancement step adds data- and control-flow context from the source (via Tree-sitter and Joern) to patches with few changed statements. An LLM then locates the pseudocode slice in the IDA Pro decompilation of the target function that corresponds to each patch slice. Finally, matched slices are normalized into equations and checked for equivalence with the Z3 SMT solver, and the LLM judges the cases Z3 cannot model, deciding whether the function matches the pre-patch or the patched version. On 4,480 test cases from 73 CVEs in OpenSSL, FreeType, tcpdump and libxml2, compiled with GCC 9.4.0 and Clang 6 at O0–O3 for x86, x64 and ARM, the authors report an average F1 of 0.77 on x86 with Claude 3.5 Sonnet, against 0.67 for Robin and 0.68 for PS3, and F1 from 0.71 to 0.77 across five compiler–architecture settings.
Lares answers a narrow question: given a specific CVE patch diff, the project source with its history and a candidate function, is that function pre-patch or patched? It needs both the patch diff and the project source, and assumes a separate binary code similarity tool (the paper names BinaryAI as an example) has already located the candidate function. It does not find vulnerable functions, discover new vulnerabilities or diff arbitrary binaries. Its judgments rest on an LLM plus SMT reasoning over decompiler pseudocode: Z3 handles only about 21% of test cases (Z3-only verification has recall 0.221), so most decisions come from the LLM, and the authors list hallucinated comparisons, stealthy edit-type patches and repeated patch statements as failure modes. Results use IDA Pro 7.5 and a commercial model at about $0.20 per case; GPT-3.5-Turbo and DeepSeek-Coder score lower. The evaluation covers four C projects, 73 CVEs and binaries the authors compiled themselves, plus 20 vulnerabilities disclosed after October 2024 (400 cases) to check training-data contamination; firmware, other languages and real vendor builds are not evaluated. The repository has no license file, so no reuse terms are granted.
2025NeurIPS 2025 Datasets and BenchmarksConference paper
Introduces two million C/C++ binary-source function pairs using a Compile-Trace-Filter pipeline, with an evaluation suite spanning translated HumanEval/MBPP tasks and recent GitHub functions.
This is function-level decompilation. Re-execution tests cover HumanEval/MBPP; GitHub2025 provides no unit tests. Inlining can prevent one-to-one source correspondence, and the fine-tuning study uses 10% of the corpus. These results do not establish whole-program recovery.
SymGen infers function names for stripped binaries by generating them with a LoRA-fine-tuned Code Llama model instead of classifying over a fixed name vocabulary. Ghidra decompiles each stripped function, its own name is masked, and callee names are normalized for deduplication. Before fine-tuning, the model is domain-adapted on decompiled code paired with LLM-generated summaries of the matching source functions. On functions from 33 C projects built for x86-64, x86-32, ARM and MIPS at O0–O3, deduplicated from 2,237,915 to 495,494, the authors report average word-level F1 of 0.277 (0.380 on x86-64), plus case studies on obfuscated binaries and Linux malware.
Names are split into words and scored with precision, recall and F1, with SymLM’s CodeWordNet grouping similar words; predictions are descriptive guesses, not recovered original symbols. Average F1 is 0.277 and lower on MIPS (0.137) and ARM (0.262). The gains of up to 409.3%, 553.5% and 489.4% in the abstract are author-reported relative improvements over AsmDepictor; the baselines were retrained on the authors’ deduplicated dataset, where they score far below their own papers, and XFL was compared on x86-64 only. A separate model is trained for each architecture and optimization level. All binaries were built with GCC 9.4.0, inputs depend on Ghidra output, and Code Llama’s closed pretraining data leaves leakage possible. Obfuscation tests apply three default Hikari passes to 363 binaries from six projects (F1 0.271–0.303); the malware study covers 812 functions in 16 x86-64 builds (F1 0.348).
Combines two fine-tuned language models with Prolog-based cross-function reasoning to infer local-variable symbols and user-defined data structures from stripped-binary decompilation.
The method depends on underlying decompilation and learned predictions over C/C++ data. Cross-checking reduces conflicting predictions but does not prove that recovered names or layouts match the unavailable source. The public artifact includes models, processed data, and evaluation code.
Authors & citation
Danning Xie; Zhuo Zhang; Nan Jiang; Xiangzhe Xu; Lin Tan; Xiangyu Zhang
Adds two techniques to LLM decompilation of function-level x86-64 GCC assembly into C. Self-Constructed Context Decompilation (sc²dec) recompiles a model’s first compilable output and uses the resulting assembly–source pair as an in-context example for a second attempt, without fine-tuning. Fine-grained Alignment Enhancement (FAE) fine-tunes llm4decompile-6.7b with LoRA on statement-level assembly–C alignments derived from DWARF debug information. Combining both, the authors report 52.41% average re-executability on Decompile-Eval, against 48.60% for the base model.
Results are function-level re-executability on Decompile-Eval: 164 HumanEval-derived C problems compiled at O0–O3 and checked by recompilation and HumanEval test cases. Passing these tests is not whole-program or all-input equivalence, and the paper does not evaluate real-world binaries, other architectures or readability. sc²dec helps only when the first output recompiles, so low re-compilability limits its gains; the authors report 1–2% lower results when context is built at a mismatched optimization level, and clang-built contexts scored slightly below GCC. FAE used 10,000 ExeBench functions limited to the standard C library, one training epoch and a single 6.7B base model; the authors did not validate it on larger datasets or models, and it slightly lowered the 1-shot result. GPT-4 and DeepSeek baselines are API snapshots that may not be reproducible.
Authors & citation
Yunlong Feng; Dechuan Teng; Yang Xu; Honglin Mu; Xiao Xu; Libo Qin; Qingfu Zhu; Wanxiang Che
Trains language models for assembly-to-C decompilation and for refining Ghidra pseudocode, evaluating whether recovered functions can be recompiled and executed correctly.
The paper limits its experiments to C on x86 and function-level benchmarks. Re-executability on sampled tests does not establish whole-program equivalence or recovery of the original source. Later repository releases are separate from the 2024 paper.
WaDec decompiles WebAssembly text (wat) into C by fine-tuning CodeLlama-7b with LoRA on loop-level wat–C snippet pairs. Functions are ordered by dependency and sliced so that each input holds at most one loop, with nested loops replaced by markers. Prompts add declarations of called functions, previously defined local variables and offset-to-string mappings; the outputs are then reassembled, with string constants restored and variables renamed. Training data come from 51,768 the-stack-dedup and 377 TheAlgorithms/C programs compiled with Emscripten. On 3,977 C files, the authors report 3.34% code inflation, 52.11% recompilability, 43.55% re-execution and 27.15% output consistency, compared with Wasm2c and Ghidra.
The reported rates are author results on C programs compiled to Wasm with Emscripten. About half of the outputs recompile, fewer than half re-execute and roughly a quarter reproduce the original output, so decompiled code is generally not semantically equivalent. The 3,977-file benchmark comes from the same two sources as the training programs. Optimized builds are not evaluated: the authors neither adapted the method to nor trained it on O1/O2 code. The model also handles data structures poorly, often flattening them into separate variables or arrays. Only C sources are studied; Wasm produced from other languages or toolchains is not evaluated. Baselines are Wasm2c and Ghidra's Wasm plugin, and LLM baselines were excluded because none could decompile Wasm functions without fine-tuning. Slicing means one function may be decompiled several times, at about 0.1 to 2 seconds per prompt.
Benchmarks 16 code and general LLMs on function name recovery and summarization of stripped binaries. The benchmark takes 12 popular C projects built for x86-64, strips and decompiles 28 binaries with IDA Pro, aligns 81,185 functions with source through DWARF, and keeps 2,000 functions after expert review. Zero-shot LLMs are compared with SymLM, NER, BinT5 and HexT5. Code Llama 34B leads name recovery with token-level F1 of 27.59, and ChatGPT leads summarization with BLEU-4 7.37. The study also measures few-shot prompting, LoRA fine-tuning of two 7B models on GNU binaries, and a case study on a stripped open-source Linux remote-access trojan.
The benchmark covers x86-64 builds with each project’s default compiler and optimization settings and IDA Pro pseudocode only, so the results do not establish behavior on other architectures, optimization levels, decompilers or obfuscated code, which the authors exclude. Reference summaries were generated by ChatGPT from source and screened by three experts, and ChatGPT is also an evaluated model. BLEU-4, METEOR and Rouge-L measure overlap with those references, which the authors say may not fit binary summarization. Name recovery uses token-level precision, recall and F1 rather than exact matches. Fine-tuning used 30,000 functions from O0 x86-64 GNU builds. Results describe model versions released between mid-2023 and early 2024, including gpt-3.5-turbo-16k.
CLAP (Contrastive Language-Assembly Pre-training) learns assembly-function embeddings aligned with natural-language explanations. A dataset engine compiles Ubuntu C/C++ packages with GCC and Clang at O0–O3 and Os, has GPT-3.5 explain about 2.6 million matching source functions, and fine-tunes a Vicuna-13B “shadow model” on half of those explanations to label more code, yielding 195 million assembly–explanation pairs. A 110M-parameter assembly encoder, using address rebasing, a WordPiece tokenizer and jump-aware embeddings, is first pre-trained on assembly and then contrastively aligned with a text encoder. The embeddings support zero-shot classification by matching code against text prompts. On BinaryCorp-3M similarity search in pools of 10,000 functions, zero-shot CLAP reports 0.836 average recall@1 against 0.571 for fine-tuned jTrans.
Training explanations are generated by GPT-3.5 and a fine-tuned Vicuna model from source code, not written by humans; five experts scored 50 explanations per model, mostly as acceptable. The authors name the Ubuntu-only build environment and the GPT-3.5 explanations as possible sources of bias. Functions are disassembled with IDA Pro 7.6, and functions with fewer than three basic blocks are excluded. Similarity results use BinaryCorp-3M optimization-level pairs, where baselines get a linear probe and jTrans is also fully fine-tuned while CLAP is not; crypto identification (17 algorithms, about 70K functions labeled by name and package metadata) and protocol categorization (18 types) use datasets the authors built from Ubuntu packages, where fine-tuned CLAP beats the baselines while zero-shot CLAP trails the best baseline on crypto. The malware and sorting results are single case studies. Training used 32 A100 GPUs. CLAP produces embeddings and label scores; it is not a decompiler or a malware detector.
Authors & citation
Hao Wang; Zeyu Gao; Chao Zhang; Zihan Sha; Mingyang Sun; Yuchen Zhou; Wenyu Zhu; Wenju Sun; Han Qiu; Xi Xiao
TYGR infers the types of stack and heap variables in stripped binaries by running an architecture-agnostic, intra-procedural data-flow analysis on angr's VEX IR and classifying each variable's data-flow graph with a graph neural network. It predicts base types, pointers and struct member layouts, covering 97.1% of the types observed in its data. The authors also build TyDa, 163,643 binaries compiled with debug information from Gentoo and Debian C packages for x86, x64, AArch64, Arm32 and MIPS at O0–O3, and deduplicate functions in the evaluated subset before splitting it. On x64 across O0–O3 they report 76.6% overall and 45.2% struct accuracy, and say TYGR beats prior work by at least 26.1% overall and 10.2% on structs.
Ground truth comes from DWARF information in binaries compiled by the authors, and training and testing use TyDa-min, about 8% of TyDa's binaries, so the results do not cover wild, obfuscated or C++ code. The analysis is intra-procedural and covers stack and heap variables only, not globals; unresolved indirect jumps can leave code unexplored, and adjacent stack structs are predicted as one struct, though the authors find this rare in TyDa. Accuracy plateaus near 80% as training data grows, which the authors attribute to rare types and types that usage alone cannot distinguish. Comparisons follow each baseline's original setting: DIRTY and OSPREY on x64 only, and OSPREY through results its authors supplied for Coreutils x64-O0. Accuracy is not compared directly with Ghidra or IDA Pro. Predictions choose from a fixed set of base, pointer and struct-member types; they do not recover type names or full struct declarations.
Authors & citation
Chang Zhu; Ziyang Li; Anton Xue; Ati Priya Bajaj; Wil Gibbs; Yibo Liu; Rajeev Alur; Tiffany Bao; Hanjun Dai; Adam Doupé; Mayur Naik; Yan Shoshitaishvili; Ruoyu Wang; Aravind Machiry
Tests decompilers function by function on binaries without needing source code. RECOMPILER repairs decompiled C so individual functions recompile, SYMDIFF compares symbolic models of the original binary function and the recompiled decompiled function using patched angr and PROMPT symbolic executors, and TUNER toggles decompiler heuristic rules to locate, and sometimes work around, the rule behind an inaccuracy. On Ghidra 10.0 and angr 9.2.30, 2,004 binaries and object files from coreutils, util-linux and six popular GitHub C projects yield 93K decompiled functions; the authors report 4,515 incorrectly decompiled functions, 8 reproduced known bugs, 17 previously unknown bugs and 7 bugs fixed automatically.
Only Ghidra and angr were tested; Hex-Rays, Binary Ninja, RetDec and LLM-based decompilers were not, and the authors estimate that supporting a decompiler with an unsupported IR would take at least 40 days of symbolic-engine work. Results are function-level symbolic equivalence checks on the authors’ 2,004 x86-64 Linux inputs, built with GCC 11.1.0 and Clang 16.0.0 at each project’s default optimization and excluding binaries over 30 MiB. Coverage is partial: RECOMPILER recompiled 72.4% of Ghidra’s and 45.0% of angr’s decompiled functions, and SYMDIFF analyzed 91.3% and 93.9% of those. Symbolic execution bounds loops, uses a two-minute default timeout, treats global variables as constants and does not parse floating-point constraints; accuracy trade-offs cause false positives, measured on 100-function samples per decompiler. TUNER automatically fixed 72 of the 443 inaccuracies it examined. A function passing the check is not proof of whole-program correctness.
Authors & citation
Muqi Zou; Arslan Khan; Ruoyu Wu; Han Gao; Antonio Bianchi; Dave (Jing) Tian
HermesSim detects binary code similarity by lifting each function to Ghidra P-code and building a semantics-oriented graph that exposes operand structure inside instructions, def-use, control-flow and effect-order relations, while dropping semantics-independent details such as which register temporarily holds a value. A gated graph neural network with a multi-head softmax aggregator embeds the graph, and functions are compared by embedding similarity. On the Dataset-1 benchmark of Marcelli et al., the authors report 48.1% RECALL@1 for x64 cross-compiler retrieval in pools of 10,000 functions, against 31.4% for a fine-tuned jTrans, with a 388K-parameter model.
Laboratory results reuse Dataset-1 from Marcelli et al.: 257K, 13K and 522K training, validation and test functions for x86, ARM and MIPS in 32 and 64 bits, built at O0–O3 and Os by GCC and Clang. HermesSim keeps that dataset's IDA Pro 7.3 control flow and basic-block boundaries and uses Ghidra only to disassemble and lift each block, so the results do not measure function or CFG recovery. Each setting samples 1,000 queries against pools of up to 10,000 functions, and the gains are relative to the paper's own runs of SAFE, Asm2Vec, GMN, Trex and jTrans, which supports x86 only and was compared on the x64 subset. The real-world study searches 62,605 ARM32 and MIPS32 functions from 12 RTOS firmware images of three vendors for 10 functions linked to 5 CVEs; its ground truth comes from manually examining the top 20 results of the compared methods. String, integer and external-symbol references are not encoded, only memory effects are modeled, and unresolved indirect branches leave graphs incomplete. Similarity ranks candidates for review; it does not establish equivalence or vulnerability.
Open-codes decompiled/original C function pairs to classify how decompiler output departs from the source beyond missing names and types. Using a statement-alignment abstraction and thematic analysis, the authors build a hierarchical codebook of 15 top-level categories and 52 codes, covering correctness issues such as nonequivalent expressions and incorrect return behavior and readability issues such as extraneous variables, non-idiomatic control flow and decompiler-specific macros. The codebook, developed on Hex-Rays output, is applied to Ghidra, retdec and angr, and the authors suggest type prediction, rule-based fixes and whole-program learned techniques to address classes of defects.
A qualitative taxonomy, not a benchmark score or decompiler ranking. Coding centres on a random sample of 200 Hex-Rays decompiled/original pairs from the small-function set plus ten large functions; the authors report 235 coded pairs in total. Ghidra, retdec and angr were compared with Hex-Rays only on a 25-function subsample. Inputs are Hex-Rays decompilations of GHCC builds of majority-C GitHub repositories, biased toward projects with recognizable, working build scripts and filtered by a step that also removed some valid functions; the dataset contains no obfuscated binaries and is unlikely to include malware. Missing variable names and types were deliberately not labeled. The Java check applies the codebook to 25 Defects4J functions decompiled with JD-GUI as a side study. The issues’ effect on reverse-engineer productivity is left to future work.
Authors & citation
Luke Dramko; Jeremy Lacomis; Edward J. Schwartz; Bogdan Vasilescu; Claire Le Goues
2024Proceedings of the ACM on Software Engineering (FSE 2024)Journal article
CodeArt pre-trains a BERT-sized encoder from scratch on 26 million stripped x86-64 functions built from jTrans's BinaryCorp. Instead of relying on symbols, it uses IDA Pro control-flow graphs and a binary data-flow analysis to compute control and data dependences, then restricts self-attention with masks so a token attends only to instructions it transitively depends on or that depend on it, to tokens in its own instruction, and to a global [CLS] token; pre-training masks tokens and perturbs the dependence graph. The authors fine-tune it for binary similarity, malware family classification and type inference. They report improvements over prior models from 53% to 64% recall@1 for similarity in pools of 500, from 49% to 60% LRAP for malware families, and from 74% to 94% average F1 for type inference.
The model handles x86-64 only, with functions disassembled by IDA Pro and inputs capped at 512 tokens. Similarity is fine-tuned on BinaryCorp-3M and tested on six projects (Curl, Binutils, ImageMagick, SQLite, OpenSSL and PuTTY), with Coreutils for validation, against GNN, GMN, jTrans and DiEmph baselines; on PuTTY, whose functions are short, CodeArt is only comparable to DiEmph. Type inference uses the x86-64 part of StateFormer's dataset with 35 labels and compares with StateFormer's reported results rather than a rerun. Malware classification uses 5,484 VirusTotal and VirusShare samples from the 100 most frequent families, against jTrans and a re-implemented CNN. All gains are relative to these baselines and settings. CodeArt produces embeddings and task-specific predictions; it is neither a decompiler nor a malware detector.
StackSight decompiles WebAssembly text-format functions into C++ with LLM prompting. A static analysis annotates each WAT instruction with the virtual-stack state, using symbolic notation where branches diverge; a one-shot chain-of-thought prompt then asks the model to recover variable meanings, summarize the function and write C++. With GPT-4 the share of outputs passing all benchmark tests rises from 19.7% to 36.5% on HumanEval-X and from 42.4% to 59.6% on MBXP relative to one-shot prompting; GPT-3.5 and Code Llama-7B-Instruct are also evaluated.
Inputs are short exercise functions from the C++ splits of HumanEval-X (161 of 164 compiled) and MBXP (773 of 974), built as Emscripten 3.1.46 side modules at -Oz with function names kept by --profiling-funcs and converted with wasm2wat 1.0.33; minified, obfuscated or whole-application WebAssembly is not evaluated. Correctness is the share of outputs that compile and pass the benchmarks' test cases after a test oracle renames functions, inserts tests and adds namespace directives, with manual pass-by-value edits. Passing these tests is not proof of equivalence with the original module. Summaries are scored by BERTScore and SBERT; the user study had 15 participants on 20 snippets per dataset. wasm2c and an earlier neural decompiler were excluded for producing no compilable output, and results depend on specific hosted model versions.
Authors & citation
Weike Fang; Zhejian Zhou; Junzhou He; Weihang Wang
STRIDE predicts the names and types of variables in decompiler output without a neural model. From training decompilation with known labels, it builds a database mapping the token sequences (N-grams, with N from 2 to 60) immediately left and right of each variable occurrence to the five most frequent names or types. Variable placeholders are canonicalized, and address-like tokens and large constants are bucketed so that N-grams can match. At prediction time it looks up the longest matching N-gram on each side of every occurrence, scores the returned labels by how specific the match is, and sums the scores across the function. The prototype is about 1,000 lines of Python and runs on a CPU. On DIRT functions not seen in training, the authors report 66.4% retyping accuracy, against 52.3% for DIRTY with beam search, and 56.2% renaming accuracy, against 51.3% reported for VarBERT. Across 16 VarCorpus splits, renaming accuracy is higher than VarBERT on 11 and 1.55 points higher on average. Prediction takes 8.2 ms per function on a CPU, against 43.7 ms for VarBERT on an NVIDIA L4 GPU.
An arXiv preprint with no stated venue. STRIDE classifies by exact N-gram matches, so its predictions depend on token sequences seen in the training data: it cannot fuzzy-match or predict a name or type absent from training, and a model trained on one decompiler's token format is not expected to transfer to another. Results are reported on decompiler output from the DIRT, DIRE and VarCorpus benchmark datasets. DIRT pairs Hex-Rays output for GCC -O0 builds of GitHub projects with DWARF-derived labels, the older DIRE dataset was prepared similarly, and VarCorpus covers Gentoo packages decompiled with Hex-Rays and Ghidra at -O0 to -O3. Retyping is evaluated only on DIRT, and only x86 C/C++ code is covered. Accuracy is exact match, so near-synonyms count as errors. About 60% of DIRT test functions also appear in training, where STRIDE reaches about 99% by memorization; the not-in-train figures are the more representative ones. DIRT keeps function and global names, and removing them lowers renaming accuracy by a few points. Several baseline numbers are copied from earlier publications rather than rerun. The authors' README expects poor results on custom closed-source binaries with structures absent from training. The repository is a research prototype with no decompiler plugin or trained databases.
Authors & citation
Harrison Green; Edward J. Schwartz; Claire Le Goues; Bogdan Vasilescu
The paper evaluates GPT-4 for binary reverse engineering through interpretation of human-written and decompiled code, from basic code explanation to malware analysis.
The paper evaluates four LLMs on deobfuscating real-world malicious scripts from the Emotet malware campaign and examines their use in reverse engineering.
Studies binary function similarity when function inlining makes one binary function correspond to several source functions. From BinKit builds of 51 GNU projects with and without -fno-inline (9 GCC and Clang versions, O0–O3, six x86, ARM and MIPS targets; 216 combinations each), debug-line mappings label 5,621,140 cross-inlining pairs. The authors identify Leaf-, Root- and Internal-Inlining patterns, show that Gemini, SAFE and GMN lose accuracy on them, and propose CI-Detector, which trains one attributed-CFG graph neural network per pattern and aggregates their scores, reporting 81% precision and 97% recall.
The three patterns are derived from the authors’ own matrix of 51 GNU projects, and the authors note that other compilation settings, such as non-default optimizations, or projects with different inlining conventions may not fit the learned models. Labels depend on IDA Pro and DWARF debug tools, which the authors list as threats to construct validity, and ACFGs come from IDA Pro 7.3, so disassembly errors can affect results. Precision and recall come from balanced sets of 40,000 positive and 40,000 negative test pairs with a threshold tuned for F1, not from large-pool search. Baselines are released Gemini, SAFE and GMN models plus Bingo-style inlining strategies with Asm2Vec. Obfuscated binaries and LLM-based similarity models are not evaluated.
Authors & citation
Ang Jia; Ming Fan; Xi Xu; Wuxia Jin; Haijun Wang; Ting Liu
BinaryAI identifies which open-source C/C++ libraries a stripped binary reuses by matching its functions to source code. Ghidra decompiles each binary function to pseudo-C, and a Pythia-410M model, trained with a CLIP-style contrastive loss on about 10 million binary–source function pairs, embeds pseudo-code and source functions in one space. Candidates retrieved from a corpus of 56,342,179 source functions from 12,013 libraries are refined with link-time locality and the call graph, and a library is reported when the share of its matched functions exceeds a threshold. The authors report 22.54% recall@1 against 10.75% for CodeCMR, and library-detection precision and recall of 85.84% and 64.98% against 73.36% and 59.81% for an unnamed commercial tool.
Training pairs come from ArchLinux and AUR packages compiled automatically with DWARF debug information, and the corpus and test set cover C/C++ projects only. The SCA test set is 150 stripped binaries the authors compiled from 85 GitHub projects, with 1,045 reused libraries labeled manually from build files, headers and license metadata; function-level ground truth covers 15 of those binaries (23,529 pairs). The recall@1 figures combine the model's validation set with those 15 binaries. The two commercial baselines are anonymized under confidentiality agreements, and every baseline was rebuilt on the authors' 12,013-library dataset. The 112 false positives the authors examined all trace to incomplete library-dependency data. The model, training pairs, source-function corpus and labeled test set are not public, and the hosted service the paper links may run a different model and corpus from the evaluated 2024 system. Matches identify reused source; they are not a decompilation or proof that the binary is unmodified.
Authors & citation
Ling Jiang; Junwen An; Huihui Huang; Qiyi Tang; Sen Nie; Shi Wu; Yuqun Zhang
FoC uses FoC-BinLLM to summarize cryptographic functions in stripped binaries and FoC-Sim to retrieve similar implementations. It evaluates both models and applications to virus analysis and 1-day vulnerability detection.
Evaluation uses a constructed cryptographic binary dataset and automatically generated semantic labels. It reports a 14.61% higher ROUGE-L score than ChatGPT for FoC-BinLLM and 52% higher Recall@1 than previous best methods for FoC-Sim.
SLaDe translates function-level assembly into C with a 200M-parameter encoder–decoder model, code-specific tokenization and PsycheC type completion. It studies x86 and ARM at -O0 and -O3, combining generated candidates with input/output checks. The input is assembly text, not raw machine bytes; disassembly is outside the method.
Evaluation covers about 4,000 ExeBench functions and a 112-program synthesis suite. Beam size 5 selects the first candidate passing finite I/O tests; generated functions are inserted into the original calling context. Passing these tests is not whole-program or all-input equivalence. Edit similarity is a readability proxy. Longer functions, external calls and incorrect type/operator predictions remain failure modes; the model has 1,024-position source and target windows. Historical baseline comparisons have different coverage and do not establish current rankings.
Authors & citation
Jordi Armengol-Estapé; Jackson Woodruff; Chris Cummins; Michael F. P. O'Boyle
CEBin combines code embeddings to shortlist candidate functions with pairwise comparison to detect similar binary code, including vulnerable functions.
The paper evaluates CEBin on three binary code similarity datasets and a large-scale benchmark for 1-day vulnerability detection. It reports finding similar functions among millions of candidates in seconds, with 85.46% recall on the vulnerability task.
Authors & citation
Hao Wang; Zeyu Gao; Chao Zhang; Mingyang Sun; Yuchen Zhou; Han Qiu; Xi Xiao
VarBERT suggests variable names for IDA Hex-Rays and Ghidra decompiler output with a BERT-style model pre-trained on 5.2M human-written C functions from Debian and fine-tuned on decompiled code; it also predicts whether each variable came from the developer or was introduced by the decompiler. The authors build VarCorpus from Gentoo C and C++ packages compiled for x86-64 at O0–O3, labeling decompiled variables from debug symbols. On per-function splits of O2 binaries, top-1 accuracy is 54.43% for IDA and 54.49% for Ghidra; on an IDA-O0 subset it reaches 50.70%, against 35.94% for DIRE and 38.00% for DIRTY.
Predicted names are learned suggestions scored against developer names, not recovered originals. Top-1 accuracy drops to 42.40% (IDA) and 40.49% (Ghidra) on per-binary O2 splits, and the per-binary IDA-O0 lead over DIRTY is 4.52 points. The DIRE and DIRTY comparison covers IDA at O0 only, on about 400K functions (roughly 15% of VarCorpus), because DIRE's training time was prohibitive. The method can name only variables the decompiler recovers, and labeling relies on IDA and Ghidra preserving variables when matching debug symbols. Corpora come from Linux distribution packages, mainly x86-64 with a smaller AArch64 O2 experiment, and may not represent Windows or macOS executables; deduplication removes exact duplicates only. The user study had 27 students who completed 13 function pairs. Types, structures and recompilable code are outside its scope.
Authors & citation
Kuntal Kumar Pal; Ati Priya Bajaj; Pratyay Banerjee; Audrey Dutcher; Mutsumi Nakamura; Zion Leonahenahe Basque; Himanshu Gupta; Saurabh Arjun Sawant; Ujjwala Anantheswaran; Yan Shoshitaishvili; Adam Doupé; Chitta Baral; Ruoyu Wang
Refines existing C-like decompiler output through three coordinated roles that select, propose and check changes. It simplifies structures, adds comments and renames variables; Micro Snippet Semantic Calculation compares simulated symbol values and calls before accepting structural changes. The authors evaluate readability and naming on 620 functions from four codebases, with separate stripped and non-stripped variants.
The reported 24.4% reduction is a Halstead-effort proxy on non-stripped output, not a measured reduction in human effort. The 62.9% comment result is the average non-trivial share among manually judged correct comments. MSSC is an intraprocedural heuristic: calls are abstracted and loops become branches; parsing, aliasing and path growth remain limitations. Its 80-case manual audit reports 84% accuracy and 85% recall, including three harmful changes missed. This supports assisted pseudocode comprehension, not guaranteed binary equivalence, recompilable whole programs or original-source recovery.
2024NeurIPS 2024 Datasets and BenchmarksConference paper
Assemblage is a distributed system that crawls GitHub and vcpkg, builds C/C++ projects under many compiler configurations, and records source-to-binary mappings such as function addresses, source code, comments and PDB or DWARF debug information. Run on AWS for about a year, it produced 890k Windows PE binaries built with Visual Studio 2015–2022 across optimization levels, architectures and Debug/Release modes, plus 428k Linux ELF binaries from GCC and Clang, in 29 configurations. Case studies use the data for compiler provenance with PassTell, function similarity with a graph neural network and jTrans, and function-boundary identification comparing XDA with IDA.
The binaries are builds of open-source GitHub and vcpkg projects, many of them small student projects or utilities, so they do not represent commercial, packed or obfuscated software. Assemblage does not independently scan for malware; the authors rely on GitHub’s policy against malware and plan scanning for a later version. The reported build success rate was about 30% on Windows and 7% on Linux. Public datasets include only repositories with permissive licenses, so the downloadable copy is a subset of the paper’s corpus, and repository links in the metadata tie binaries to authors’ GitHub profiles. The case studies retrain existing models under their original protocols with minimal changes; they show Windows/Linux generalization gaps, such as jTrans MRR of 0.52 on Windows against 0.83 on Linux, rather than new analysis methods.
Authors & citation
Chang Liu; Rebecca Saul; Yihao Sun; Edward Raff; Maya Fuchs; Townsend Southard Pantano; James Holt; Kristopher Micinski
BinSum benchmarks LLMs on summarizing binary functions in natural language. The authors compile 44 GNU projects with GCC 7.5 for x86, x64, ARM and MIPS at -O0 to -O3, match 557,664 binary functions to developer comments as reference summaries, and present each function as raw bytes, assembly, IDA microcode or decompiled code with and without symbols. GPT-4, ChatGPT, Llama 2 and Code Llama receive prompts selected from 320 GPT-4-generated candidates, with the fine-tuned BinT5 as a baseline; outputs are scored by embedding similarity alongside BLEU, METEOR and ROUGE-L. Decompiled code is the most useful representation, and stripping symbols lowers the average similarity score from 0.449 to 0.202.
Reference summaries are developer comments extracted from source, which the authors note may be inconsistent with the code. Embedding similarity with all-mpnet-base-v2 and n-gram metrics measure closeness to those comments, not whether a summary is correct or useful to an analyst, and no human evaluation is reported. All binaries are open-source GNU programs built with one GCC version, and the authors note that firmware, malware and obfuscated code are not covered. Prompts were chosen on 1,000 randomly sampled functions. The results describe the June 2023 gpt-4-0613 and gpt-3.5-turbo-16k-0613 snapshots and the 7B and 13B Llama 2 and Code Llama models, so they do not rank current models.
Authors & citation
Xin Jin; Jonathan Larson; Weiwei Yang; Zhiqiang Lin
The paper proposes a two-step hybrid LLM approach to refine IDA-Pro decompiled C code for recompilation. On popular C test cases, it reports a recompilation success rate above 75% with moderate effort.
kTrans uses a knowledge-aware Transformer to embed binary code. It evaluates the embeddings through outlier detection and visualization, and on binary code similarity, function type recovery, and indirect call recognition.
Covers binary code and three downstream tasks: binary code similarity detection, function type recovery and indirect call recognition, with reported improvements of 5.2%, 6.8% and 12.6%.
BTD (Bin to DNN) decompiles stripped 64-bit x86 executables emitted by deep-learning compilers into model specifications: operator types, network topology, dimensions and parameters. A learned classifier labels disassembled functions as DNN operators, Intel Pin dynamic analysis chains them, and taint-reduced, trace-based symbolic execution with per-operator patterns recovers dimensions and parameter layouts. The authors rebuild recovered models in PyTorch and recompile them to compare behaviour with the original executable.
Evaluated only on 64-bit x86 executables from TVM v0.7, v0.8 and v0.9.dev (-O0 and -O3), three Glow snapshots from 2020–2022 and NNFusion v0.2/v0.3; GPU, accelerator and other-architecture executables are discussed, not decompiled. It assumes flawless disassembly and function recovery (IDA Pro 7.5 in the paper) and known compilation provenance, and dimension/parameter patterns are hand-designed per operator: 22 of 36 ONNX operators needing such patterns are covered, excluding quantized operators such as QLinearConv. The authors report correct specifications for 59 of 65 executables from seven CV models and FastText; one TVM -O3 ResNet18 Conv layout required manual fixing with ground truth, and NNFusion compiled only its own VGG11. Recompiled behaviour is compared on 100 ImageNet images or 50 crafted FastText inputs, not proven for all inputs. Obfuscated executables were not evaluated.
Authors & citation
Zhibo Liu; Yuanyuan Yuan; Shuai Wang; Xiaofei Xie; Lei Ma
XFL treats function naming in stripped binaries as extreme multi-label classification: names are split into tokens, and PfastreXML ranks which tokens from a vocabulary of frequent labels fit each function. Its DEXTER embedding combines static-analysis features built on angr’s VEX IR with call-graph and whole-binary context, and a separate language model orders predicted tokens into a name string. On 741,724 functions from 10,047 x86-64 C binaries in Debian packages, with a 1,024-label vocabulary, the authors report precision 0.835, recall 0.575 and F1 0.681, ahead of Punstrip, Debin and Nero in precision and F1.
Predictions are token labels from a fixed vocabulary, ranked and optionally assembled into names; they are not recovered source names, and scoring matches tokens rather than exact names. Evaluation uses x86-64 ELF C binaries from Debian packages, built with mixed compilers and optimization settings, plus Nero’s smaller dataset; the trained model is specific to x86-64 GNU/Linux binaries. All tools receive function boundaries from symbol tables at test time, so boundary-detection errors are excluded. Many test names also occur in training; restricted to names never seen in training, F1 falls to 0.242. Obfuscated and adversarial binaries are out of scope. Debin and Dire figures on the Nero dataset are copied from the Nero paper rather than rerun.
Authors & citation
James Patrick-Evans; Moritz Dannehl; Johannes Kinder
2023IEEE Transactions on Software EngineeringJournal article
Builds BinKit, a reproducible cross-compilation benchmark of 243,128 binaries and 36,256,322 functions from 51 GNU packages, covering 1,352 combinations of eight architectures, nine GCC and Clang versions, five optimization levels, and PIE, LTO, no-inline and Obfuscator-LLVM options. TikNib, an interpretable baseline over simple presemantic function features, measures how compilers, options and architectures change those features. The authors report that a few basic features approach a deep-learning tool’s accuracy, that type features push AUC above 0.99, and trace failures to IDA Pro analysis errors, compiler back-end differences and architecture-specific code.
Studies function-level, presemantic (syntactic and structural) features on Linux GNU packages compiled from source with debugging information for ground truth; semantic features, NLP-based models and whole-binary or other similarity tasks are outside the evaluation. The above-0.99 AUC uses argument and return types derived from source via Ctags and debugging information, not types recovered from stripped binaries. The state-of-the-art comparison sets TikNib’s AUC on VulSeeker’s ASE datasets against VulSeeker’s author-reported numbers rather than a re-run, and omits the original firmware images. Obfuscation covers Obfuscator-LLVM substitution, bogus control flow and flattening applied once. Experiments used IDA Pro 6.95, and the authors cannot rule out erroneous MIPS feature extraction. The results do not rank later learned or LLM-based models.
Authors & citation
Dongkwan Kim; Eunsoo Kim; Sang Kil Cha; Sooel Son; Yongdae Kim
Asteria-Pro adds domain-knowledge pre-filtration and re-ranking to Tree-LSTM-based binary code similarity detection. It evaluates firmware vulnerability search across diverse IoT hardware architectures.
The evaluation covers a firmware bug-search task: pre-filtration reduced calculation time by 96.9%; re-ranking improved MRR by 23.71% and Recall by 36.4%. A large-scale search found 1,482 vulnerable functions at 91.65% precision.
Authors & citation
Shouguo Yang; Chaopeng Dong; Yang Xiao; Yiran Cheng; Zhiqiang Shi; Zhi Li; Limin Sun
SymLM predicts function names for stripped binaries as sets of name words. It encodes each function’s instructions with a Trex model pretrained on micro-execution traces, fuses them with embeddings of callers, internal callees and external library calls taken from Ghidra’s interprocedural CFG, and scores words from a fixed vocabulary with an MLP decoder; name preprocessing and a word-embedding module, CodeWordNet, handle abbreviations and synonyms. On 1,431,169 functions from 16,027 binaries of 27 projects across x86, x64, ARM and MIPS, O0–O3 and four Hikari obfuscations, the authors report weighted average precision 0.634, recall 0.677 and F1 0.655, plus a firmware case study.
Names are predicted as unordered word sets from a fixed training vocabulary and scored with word-level precision, recall and F1, with CodeWordNet counting semantically similar words as matches. This is not exact source-name recovery, and out-of-vocabulary words cannot be produced. Binaries were built with GCC 7.5 for Linux and split by binary within the same 27 projects. On 540 binaries from 12 unseen projects, overall performance remains lower, which the authors partly attribute to vocabulary mismatch; a later NDSS 2025 comparison on deduplicated data reports SymLM’s average F1 at 0.105. NFRE comparisons are x64-only, and NERO comparisons use NERO’s dataset. Obfuscation covers four Hikari passes, not packing or encryption. In the firmware case study, 172 of 1,061 functions in eight 32-bit ARM firmware images received partially correct names.
Measures binary function similarity research by re-implementing ten representative approaches from the security, programming-language and machine-learning communities in one IDA Pro-based framework, from fuzzy hashing (Catalog1, FunctionSimSearch) to graph neural network, NLP-encoder and transformer models. Two new datasets support six tasks that vary optimization, compiler, bitness and architecture, scored by AUC and ranking metrics, plus a firmware vulnerability-search case. The authors report that fuzzy hashing fails in harder cross-architecture settings, GNN-based models perform best in almost all tasks, and many recent models score similarly on common data.
A 2022 comparison of the authors’ re-implementations, not of the original papers’ code or reported numbers, which the authors did not try to reproduce; CodeCMR was run by its own authors, and Trex was tested only on Dataset-2 with its authors’ model. Dynamic-analysis, symbolic-execution and other slow direct-comparison approaches were excluded by design, and LLM-based similarity models are not covered. Dataset-1 compiles seven open-source projects (ClamAV, Curl, Nmap, OpenSSL, Unrar, Z3, Zlib) with GCC and Clang for x86-64, ARM and MIPS in 32 and 64 bit at O0–O3 and Os with function inlining disabled, and drops functions under five basic blocks. Dataset-2 reuses ten GCC 7.5 projects released by the Trex authors. Feature extraction depends on IDA Pro 7.3; BinaryAI covered only part of the data because a recent Hex-Rays MIPS decompiler license was unavailable. The firmware case searches ten OpenSSL functions in two images. Obfuscated code is not evaluated, and the results are not a current ranking.
Authors & citation
Andrea Marcelli; Mariano Graziano; Xabier Ugarte-Pedrero; Yanick Fratantonio; Mohamad Mansouri; Davide Balzarotti
DeepDi disassembles x86 and x64 binaries by decoding a superset of candidate instructions, linking them in an Instruction Flow Graph of forward, backward and overlap relations, and classifying true instructions with a Relational Graph Convolutional Network; heuristics and a small classifier then recover function entry points. On BAP corpora, LLVM 11 built with MSVC on Windows, and SPEC CPU2006 and CPU2017, the authors report accuracy comparable or superior to IDA Pro, Binary Ninja, Ghidra, Datalog Disassembly and XDA, robustness on Linn–Debray and Hikari obfuscated binaries, single-core CPU throughput of 146 KB/s (about twice IDA Pro) and GPU throughput of 24.5 MB/s (about 350 times IDA Pro).
Covers x86 and x64 ELF and PE binaries only. ELF ground truth assumes no data in code sections and uses linear disassembly plus symbol tables; PE ground truth comes from PDB files, with data lengths estimated because PDBs give only data addresses. Results are the authors’ 2022 comparisons against Binary Ninja 2.2, IDA Pro 7.2 and Ghidra 9.1.2 on their datasets and hardware, with one GTX 2080Ti GPU; Datalog Disassembly supports only x64 ELF and so covers part of the evaluation. Obfuscation tests use Linn–Debray samples and Hikari-built binaries; the authors note that a strong adversary using the model’s gradients could construct adversarial examples, which they leave to future work. The malware case reports 10-fold cross-validation accuracy on training data and log loss from the challenge’s online judge, because test labels are not public. Disassembly accuracy does not establish correct function boundaries in every case, decompilation or source recovery.
Reframes manual reverse engineering as perfect decompilation: writing source that, compiled with the original toolchain and options, reproduces the target functions’ binary code exactly. The authors built Decomperson, a web editor that compiles, disassembles, tests and diffs each submission, and used it in Decompetition, a 24-hour Jeopardy-style competition in November 2020 with 23 unstripped challenge binaries written in C, C++, Go, Rust and Swift. Over 150 participants made more than 35,000 submissions, including over 300 perfect ones; all but one challenge was perfectly decompiled, and the recorded edit histories are analyzed to describe how humans decompile.
Participants were self-selected competitors in a single 2020 event, mostly playing solo on teams of any size, and the follow-up survey drew only 12% of registrants. The challenges are 23 small competition binaries written by the organizers, not stripped and without debug information, and Decomperson assumed the original toolchain and compilation options were known. Perfect decompilation requires matching compiler output; the authors note that it is not a traditional reverse-engineering task, does not capture skills such as triaging large binaries, and can include time spent fighting compiler idiosyncrasies. The web editor took players out of their usual tools and may have influenced timings. Participants could paste automatic-decompiler output; about 20% of C function attempts resembled tool output, mostly Hex-Rays, Ghidra or Binary Ninja. The study observes humans and does not evaluate automated decompilers or agents.
Authors & citation
Kevin Burk; Fabio Pagani; Christopher Kruegel; Giovanni Vigna
Introduces DIRTY, a Transformer model that recommends variable names and types from decompiled code and layout information, together with the DIRT corpus for training and evaluation.
The original pipeline is built around C programs and Hex-Rays output. Predicted names and types are learned hypotheses, not guaranteed original symbols. The published paper is from 2022, although its arXiv preprint first appeared in 2021.
Authors & citation
Qibin Chen; Jeremy Lacomis; Edward J. Schwartz; Claire Le Goues; Graham Neubig; Bogdan Vasilescu
jTrans embeds binary functions for similarity search with a BERT-style Transformer. Its jump-aware representation shares parameters between a direct jump’s operand token and the positional embedding of the jump target, so control flow enters the instruction sequence without a separate graph network. The model is pre-trained with masked language modeling and jump-target prediction, then fine-tuned with a cosine triplet objective on functions compiled from the same source at different optimization levels. The paper also introduces BinaryCorp, built by compiling C/C++ packages from Arch Linux’s official repositories and the AUR with gcc and g++.
The model is trained on a single architecture (x86, per the paper’s discussion); cross-architecture search is not evaluated. Only direct jumps are encoded, and indirect-jump targets are left out of scope. Disassembly and function extraction rely on IDA Pro 7.5, so boundary errors upstream are not measured. BinaryCorp pairs gcc/g++ builds across O0–O3 and Os, with train and test split by project; it does not cover obfuscation or other compilers, and only the vulnerability study adds clang variants. Author-reported Recall@1 at pool size 10,000 on BinaryCorp-26M averages 62.5% against 32.0% for the best baseline, and accuracy still falls as pools grow, which the authors note for larger real-world pools. The vulnerability search uses eight CVEs with ten variants each. Baselines date from 2022, and Asm2Vec and OrderMatters were reimplemented, so the comparison does not establish current rankings. Similarity ranking identifies candidate matches; it does not recover source code, names or types.
ExeBench is a dataset, not a decompiler. It extracts C functions from GitHub and AnghaBench, makes them compilable by locating real headers and definitions or injecting synthetic ones, and generates input/output examples from function signatures with a C++ wrapper that runs each function. After exact deduplication the authors report 4,485,817 compilable and 687,843 executable functions, of which 124,612 have rich IO that a constant or identity implementation cannot pass; the paper also describes x86 and ARM assembly at O0, Os and Ofast for each row. Decompilation studies such as SLaDe and LLM4Decompile later used it to test whether decompiled functions re-execute correctly.
IO is generated from signatures without program context: 10 pairs per function (30 across three seeds of synthetic definitions), arrays default to 32 elements, and success falls below 5% for functions with pointer–integer conversions and stays low for goto, labels and complex control flow, so executable functions skew simpler than the compilable population. Synthetic IO is produced with injected stand-in definitions rather than the functions’ real dependencies; only 4,464 functions have rich IO with real definitions. Finite IO pairs test behavior on sampled inputs, not semantic equivalence, and deduplication removes exact matches after tokenization only. Running functions with real dependencies executes untrusted code; the authors used restricted containers. The paper’s exact v1.00 data is available only on request, while the public v1.01 release has revised splits (for example 2,134 instead of 2,232 real test functions). Each function keeps its source repository’s license; ExeBench additions are MIT.
Authors & citation
Jordi Armengol-Estapé; Jackson Woodruff; Alexander Brauckmann; José Wesley de Souza Magalhães; Michael F. P. O'Boyle
2022Proceedings of the ACM on Programming Languages (OOPSLA 2022)Journal article
Elipmoc decompiles Ethereum Virtual Machine bytecode into a structured three-address intermediate representation for security analyses and reverse engineering. Built as an evolution of the Datalog-based Gigahorse decompiler, it adds transactional context sensitivity, path-sensitive inference of function arguments and returns with path merging, and fully context-sensitive reconstruction of private functions. On a uniform random sample of 5,000 unique mainnet contracts first deployed between April and September 2021, the authors report fully resolved operands for 99.5% of contracts versus 62.8% for Gigahorse, 67% more unique external-call signatures than Etherscan’s Panoramix, and over 5x faster decompilation.
Targets EVM bytecode only. The metrics are operand resolution, structured control flow, function and jump-target precision, external-call and event coverage, timeouts and speed against the 2022 Gigahorse artifact and Panoramix; they do not measure equivalence to or recovery of original Solidity source, and the output is three-address IR rather than recompilable code. Experiments used one 5,000-contract sample from a five-month 2021 deployment window, a 120-second per-contract cutoff and Souffle 2.0.2. JEB, Trustlook SECaaS and EtherSolve were not compared, and EtherVM, Porosity and Vandal were excluded as inferior or unmaintained. Reported vulnerability discoveries built on Elipmoc are described as end-to-end impact, not a systematic evaluation.
Tests whether OpenAI’s Codex model code-davinci-001 can answer reverse-engineering questions about C code in a zero-shot question-and-answer prompt. Programs come from templates the authors wrote, covering cybersecurity snippets and industrial-control algorithms such as PID controllers and filters, plus five single-file Linux malware samples from vx-underground. Each is presented as original or name-randomized source, or as Ghidra decompilation of gcc -O1 builds with debug information, without it, or stripped. After exploratory open-ended questions, generated true/false and short-answer questions about purpose, capabilities and key variables are graded against an answer oracle; Codex answered 72,754 of 136,260 questions correctly.
All results come from one model, code-davinci-001, which OpenAI shut down on 2023-03-23, so they cannot be rerun against the original model and do not describe current LLMs. Accuracy is measured with a generated true/false and short-answer quiz, paired as positive and negated questions to expose answer bias, rather than with analyst tasks; the overall 53.4% led the authors to conclude that LLMs are not yet ready for zero-shot reverse engineering. Programs are small single-file C code that fits the model’s roughly 4,000-token window, compiled only at -O1; the real-malware class has five samples. The answer oracle combines the authors’ labels with capability matches from the capa tool.
2022Workshop on Binary Analysis Research (BAR) 2022Conference paper
BTC (Beyond The C) treats decompilation as plain-text translation from compiler-emitted x86-64 assembly to source code. It uses byte-pair-encoding tokenizers trained separately on assembly and source, and a standard fairseq transformer. Function pairs are matched through the compiler's CFI and line directives rather than language-specific parsers, compiler plugins or lifters, so the same pipeline is retrained for Go, Fortran, OCaml and C. Training sets range from about 10,000 functions for small C, Go and Fortran corpora to 2 million C functions built from Debian packages. The best model, trained on the large C set, reaches a normalized average edit distance of 0.46 on held-out functions.
Evaluation covers four languages at function level and scores outputs only by normalized edit distance to the original source. No compilation, test-based or semantic-equivalence check is reported, and whole-program decompilation is not evaluated. The authors state that the method cannot guarantee semantic correctness and is not yet ready for practical use on real code: even the best score means changes to roughly half of a function's tokens. Input is compiler-produced x86-64 assembly with labels and CFI directives intact, so disassembly of stripped binaries, other architectures and compilers without such directives are outside the evaluation. Sequence-length limits exclude long functions, and the small corpora are mostly competitive-programming and small utility code. The comparison with Katz et al. cites that work's reported score on shorter snippets rather than rerunning it on the same data.
20212021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)Conference paper
ASTERIA uses a Tree-LSTM to encode function ASTs and compare binary functions across platforms. It was evaluated on function pairs and used to search IoT firmware for vulnerabilities.
The model was trained on 1,022,616 function pairs and evaluated on 95,078 pairs. Its cross-platform scope covers differing IoT architectures; a firmware search identified 75 vulnerable functions in 5,979 images.
Authors & citation
Shouguo Yang; Long Cheng; Yicheng Zeng; Zhe Lang; Hongsong Zhu; Zhiqiang Shi
2020Proceedings of the ACM on Programming Languages (OOPSLA 2020)Journal article
Nero predicts names for procedures in stripped x86-64 Linux executables; it does not produce decompiled code. Static analysis reconstructs call sites, including imported API names, and augments their arguments with concrete and abstract values along control-flow paths. LSTM, Transformer and graph neural network encoders over the resulting augmented call-site graph feed a decoder that generates the name as subtokens. On a GNU-package test set the authors report 45.53 F1 for Nero-GNN, against 35.52 for an adapted DIRE and 33.66 for Debin.
Evaluation uses 67,246 procedures from C-only GNU packages built as x86-64 Linux executables, split by package into 60,403/2,034/4,809 examples after removing C++ executables, tests, statically linked packages and duplicate package versions. Labels come from names present before stripping, and F1 is measured over name subtokens, so outputs are learned suggestions rather than recovered original symbols. Simulated API-name obfuscation, which zeroes the .dynstr section, lowers Nero-GNN to 38.83 F1; packers and heavier obfuscation are not addressed. A procedure needs at least one statically detectable call site, and inlined procedures cannot be named. Data generation relied on a licensed IDA Pro (tested with 6.95) for procedure boundaries and export, plus angr and LLVM 10, and the model code uses TensorFlow 1.13.1; no claim is made about compatibility with current tool versions.
Tests whether C decompilers preserve program behavior. The authors' DecFuzzer compiles a Csmith-generated program, decompiles the executable, substitutes the decompiled function for the original in the source, recompiles, and compares the two programs' outputs; equivalence-modulo-inputs (EMI) mutation then derives further test programs from the seeds. Across four popular decompilers the authors report 1,423 inputs that trigger decompilation errors and, after manual analysis, 13 bugs in two open-source decompilers. They argue that modern decompilers are making real progress on functional correctness, while type recovery and optimization still hinder accurate output.
The tests use small generated programs, not real-world binaries, and a failed recompilation is recorded separately from a behavioral mismatch. In the authors' artifact, test programs are compiled as 32-bit x86 with GCC at -O0, and Csmith is restricted to a single function without arrays, structs, unions, pointers or long long types, so optimized code and richer data types are outside that configuration. The four decompilers (IDA Pro, JEB3, RetDec and Radare2's r2ghidra plugin) were tested as of 2020; the error counts and bugs describe those releases, not current ones. IDA Pro and JEB3 are commercial and not included in the artifact, which sets up only RetDec and r2ghidra.
E9Patch statically rewrites x86-64 binaries by inserting jumps to trampolines without moving any other instruction, so the original set of jump targets is preserved and control-flow recovery, symbolization and debug information are not needed. Where a five-byte jump does not fit, instruction punning, padding and successor or neighbor eviction overlap the jump with existing bytes or relocate neighboring instructions into their own trampolines; physical page grouping reduces the memory and file-size cost of scattered trampolines. Instruction locations come from a separate frontend, a basic linear disassembler in the evaluation. The authors evaluate SPEC2006, Ubuntu 16.04 system binaries, and Chrome and Firefox builds over 100 MB, and demonstrate binary repair and heap-write hardening with low fat pointers.
This is static binary rewriting and instrumentation, not decompilation or source recovery. It relies on the frontend’s instruction locations and sizes being correct, assumes patched code is not read as data or self-modified, and assumes the input does not already use overlapping instructions. Each tactic can fail: 1,098 of about 10.5 million patch locations were not patched, baseline coverage alone was 72.79% for jumps and 81.63% for heap writes, and non-PIE binaries are harder than PIE ones. Chrome’s mixed code and data required starting disassembly after ChromeMain, and Firefox’s code was patched in libxul.so. Reported overheads, such as about 113% for Chrome and 46% for Firefox on Dromaeo DOM benchmarks with heap-write instrumentation, were measured with the PLDI prototype, with SPEC2006 compiled by GCC in non-PIE mode; they do not describe the current release.
RetroWrite statically rewrites 64-bit position-independent Linux binaries by generating reassembleable assembly, so instrumentation can be inserted as if at compile time. Instead of heuristics, it symbolizes references from the binary's own metadata: control-flow operands, PC-relative addressing that position-independent code must use, and data relocations become assembler labels, which separates references from scalars. On this base the authors build AFL coverage instrumentation and a binary-only AddressSanitizer. They report fuzzing throughput matching compiler instrumentation and 4.5x faster than AFL's QEMU mode, with more bugs found on LAVA-M, and binary ASan that runs 3x faster than Valgrind memcheck on SPEC CPU2006 and detects 80% more bugs in their evaluation, whose coverage tests use Juliet memory-corruption cases.
The method depends on position-independent code; the paper's implementation targets Linux x86-64 and uses the symbol table for function starts and sizes, so stripped binaries need function boundaries from another tool. It cannot safely rewrite C++ binaries with exception handling, because the DWARF unwind information is not symbolized, and it does not address obfuscated or adversarial binaries. Binary ASan instruments the stack only at stack-frame granularity and adds no redzones around globals, so it misses some overflows that compiler ASan catches, and it remains slower than compiler ASan. Results are from 2019–2020 tool versions on the authors' benchmarks. The repository's later aarch64 rewriter, described in a separate thesis, uses a different algorithm, and the paper's x86-64 results do not apply to it.
BinRec lifts an executable to LLVM IR from dynamic traces instead of static disassembly, then lowers the IR back to a recovered binary, so compiler passes can optimize, harden or otherwise transform it. Tracing runs the binary under S²E, which translates the executed code through QEMU into LLVM IR; BinRec removes the emulation instrumentation, merges traces from several inputs into one program, and adds optimizations such as register de-aliasing and promotion of emulated state to locals. When the recovered binary reaches a path that was not traced, a configurable handler stops it, falls back to the original code, or lifts the new path incrementally. The authors report recovering all SPEC CINT2006 benchmarks, including C++ programs, and apply it to reoptimization, control-flow integrity, AddressSanitizer and SafeStack hardening, and partial deobfuscation of virtualization-obfuscated code.
The prototype handles only single-threaded 32-bit x86 Linux ELF binaries and does not support self-modifying code. A recovered binary supports only control-flow paths observed during lifting, so behavior on untraced inputs depends on the chosen miss handler and on input coverage. Lifting through S²E is slow, and floating-point instructions are emulated in lifted code; across the SPEC benchmarks, recovered binaries ran at a geometric-mean 0.98x the time of unoptimized inputs and 1.29x that of optimized inputs. BinRec does not map stack slots to variables. The deobfuscation case uses two small programs obfuscated with the authors' own virtualizer, not commercial obfuscators. The McSema and rev.ng comparisons reflect 2020 versions.
Authors & citation
Anil Altinay; Joseph Nash; Taddeus Kroes; Prabhu Rajasekaran; Dixin Zhou; Adrian Dabrowski; David Gens; Yeoul Na; Stijn Volckaert; Cristiano Giuffrida; Herbert Bos; Michael Franz
2020Proceedings of the Tenth ACM Conference on Data and Application Security and PrivacyConference paper
The paper builds open datasets and tools to identify CPU architecture and endianness in binary files and object code. It evaluates classifiers on executable code sections, reporting over 98% accuracy.
Egalito statically recompiles stripped x86-64 and ARM64 binaries, using the metadata that position-independent executables and libraries carry to recover a complete disassembly in which every code and data cross-reference becomes an explicit link. Its layout-agnostic intermediate representation, EIR, stays close to machine code, so regenerated output is predictable, while a higher-level view supports analysis; data-flow analyses infer jump tables, which have no metadata. Because the original layout imposes no constraints, tools can move or rewrite code freely; the authors build nine, including continuous code randomization in which Egalito transforms itself, a retpoline defense, and software emulation of Intel CET. They report recovering the compiler's jump tables in 866 of 867 executables and libraries from 172 Debian packages, all tests passing in 90 of 149 transformed packages, and a 1.7% speedup on SPEC CPU 2006.
Egalito requires position-independent code for sufficient metadata; non-PIC inputs are outside the method except where full relocations are supplied, as in the authors' Fuchsia kernel experiment. Jump-table recovery uses data-flow analysis and an adjacency heuristic, so disassembly is not guaranteed to be complete; the one failure in the Debian study was heavily nested jump tables. The authors list obfuscated code, inline assembly that embeds data in code symbols, C++ exceptions, symbol versioning, and Go or V8 binaries among unsupported cases. Package test failures are partly attributed to these features. RISC-V support was incomplete at publication. The columbia/egalito main branch has had no commits since June 2020, so no current maintenance is implied.
Authors & citation
David Williams-King; Hidenori Kobayashi; Kent Williams-King; Graham Patterson; Frank Spano; Yu Jian Wu; Junfeng Yang; Vasileios P. Kemerlis
Combines lexical and structural information from decompiler output to predict meaningful variable names, with a training and evaluation corpus derived from compiled C projects.
This is learned identifier naming, not complete source reconstruction or an autonomous agent. Its reported results concern an x86-64 C corpus and depend on decompiler output. The original author repository is deprecated and points to a successor.
Authors & citation
Jeremy Lacomis; Pengcheng Yin; Edward J. Schwartz; Miltiadis Allamanis; Claire Le Goues; Graham Neubig; Bogdan Vasilescu
Manticore uses dynamic symbolic execution to analyze binaries and Ethereum smart contracts. The paper presents its architecture and demonstrates bug finding and code verification used for commercial clients.
2019Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and ServicesConference paper
The paper reverse engineers Broadcom Bluetooth controllers and presents InternalBlue for patching their firmware. It evaluates the framework through vulnerability tests, bug fixes, and added security features.
The work covers multiple Broadcom Bluetooth chipsets used in smartphones, wearables, and IoT devices, with experiments on their controller firmware and Bluetooth stack.
Authors & citation
Dennis Mantz; Jiska Classen; Matthias Schulz; Matthias Hollick
The paper trains a neural machine translation decompiler using a compiler. It evaluates translation from LLVM IR and x86 assembly to C, successfully decompiling over 97% and 88% of its respective benchmarks.
2019Proceedings 2019 Network and Distributed System Security SymposiumConference paper
ICSREF automates reverse engineering of PLC binaries compiled with CODESYS. It was evaluated on collected and in-house PLC binaries and used to generate an attack against a chemical process testbed.
The evaluation covers CODESYS-compiled PLC binaries from public code repositories and in-house development, plus an attack orchestrated by a commercial smartphone against a chemical process testbed.
Coda decompiles the assembly of short functions into C in two phases. An instruction-type-aware N-ary Tree-LSTM encoder and an attention-fed AST decoder first generate a code sketch; an ensemble of neural error predictors then proposes token corrections, each kept only if recompiling the candidate does not increase its edit distance to the input assembly. On synthetic benchmarks the authors report 82% average program accuracy after correction, against 12% for a sequence-to-sequence baseline with attention.
Training and test programs are randomly generated from Karel commands, math.h calls and arithmetic or logical expressions, in short and long variants (average length 15 and 30), 50,000 pairs per task, compiled without optimization: clang -O0 for the main MIPS results and gcc -O0 for an x86-64 appendix that reports sketch token accuracy only, about 6% lower on average. Real-world checks are small PyTorch C++ API model-construction and Hacker's Delight loop-free programs. Program accuracy requires an exact token match with the ground-truth source, the metric under which the paper reports 0% for conventional decompilers. Error correction assumes the compiler configuration is known, and token accuracy drops for longer programs. The results show neither optimized, whole-program nor binary-equivalent decompilation, and the paper names no author implementation.
The paper learns instruction embeddings that capture semantic relationships within and across architectures. It evaluates them on semantic basic-block comparison, outperforming a code-statistics approach.
Debin predicts debug information for stripped ELF binaries on x86, x64 and ARM. After lifting code to BAP-IR, an Extremely randomized Trees classifier decides which registers and memory offsets hold source-level variables. A conditional random field, solved with Nice2Predict, then jointly predicts names and types for variables and functions, and the result is written back as DWARF sections. On 300 held-out binaries per architecture, the authors report 68.8% precision and 68.3% recall for x64 names and types.
The dataset has 3,000 non-stripped executables and shared libraries per architecture from 830 Linux debug-symbol packages of C code built with default package settings, split 2,700/300 for training and testing. Accuracy evaluation assumes given function boundaries; for fully stripped inputs, BAP's ByteWeight infers them, which the paper cites as about 93% accurate. Debin predicts 17 types, and leaves struct and union contents, line numbers and formats such as PE outside the evaluation. A prediction counts only when it equals the original debug value. Models learned from compiler-generated, benign code may be less accurate on obfuscated binaries, hand-written assembly or malware. The 35-sample malware study is manual inspection of decompiled output with no accuracy figures, because the samples lack ground-truth debug information.
Authors & citation
Jingxuan He; Pesho Ivanov; Petar Tsankov; Veselin Raychev; Martin Vechev
rev.ng lifts binaries to LLVM IR through QEMU's tiny code generator, so one front end covers every architecture QEMU supports and the output is compilable IR. On that IR the authors build architecture-independent analyses that recover a program's control-flow graph and function boundaries without symbols or debugging information. A value-range analysis (OSRA) bounds indirect jump targets such as switch tables, and further analyses handle predicated instructions, noreturn functions, tail calls and context-dependent control flow. On coreutils built for MIPS, ARM and x86-64 with GCC and clang at -O2, -O3 and -Os, its function-boundary accuracy is close to IDA Pro and better than BAP/ByteWeight and angr.
The evaluation measures function boundaries, not the CFG directly: the authors could not collect CFG ground truth from GCC or LLVM, so boundary accuracy (Jaccard index against ELF function symbols) is presented as a lower bound on CFG quality. It covers 105 statically linked, stripped coreutils programs for Linux on three of the architectures QEMU supports; the other architectures and dynamically linked binaries were not evaluated. The prototype misses some nested-switch jump tables on x86-64 and jump-table addresses spilled to the stack on MIPS, needs more memory than the other tools on ARM and MIPS, and does not report dead code that IDA Pro finds through prologue heuristics, which the authors treat as a design choice. Baselines are IDA Pro 6.6, BAP 0.9.9 and angr as of November 2016. The paper does not decompile to C.
Authors & citation
Alessandro Di Federico; Mathias Payer; Giovanni Agosta
Ramblr produces reassembleable disassembly: assembly recovered from a stripped binary that can be edited and assembled again with an off-the-shelf assembler. Starting from angr's CFG recovery, it classifies the contents of data regions, then decides which immediates and data values are references to code or data that must become symbols. Base-pointer reattribution handles compiler-optimized references that point outside their target object, and sanity checks let Ramblr refuse to reassemble a binary it cannot handle safely, such as one that encrypts pointers. The authors evaluate 106 GNU Coreutils programs for x86 and x86-64 and 143 DARPA Cyber Grand Challenge binaries at six optimization levels. They report that Uroboros broke 22.64% of 32-bit Coreutils binaries at -O0 and 56.61% at -O1, while Ramblr broke none, and that Ramblr reassembled over 98% of CGC binaries at every level.
The prototype handles x86 and x86-64 ELF binaries produced by GCC or Clang, whose code regions contain no inline data; binaries from compilers such as MSVC, which place jump tables inline, need further work. The authors stress that static content classification is infeasible in general: Ramblr is an empirical approach that is likely to fail if, for example, a binary is linked at a base address chosen so its addresses collide with ordinary constants. The Uroboros comparison covers only 32-bit binaries and, on Coreutils, only -O0 and -O1, because Uroboros failed on 64-bit inputs and its failure rate above -O1 was not meaningful. Correctness is measured by symbolization against linker ground truth and by each program's test cases, which do not prove full equivalence. Results are from 2017 versions of angr and the evaluated toolchains.
Authors & citation
Ruoyu Wang; Yan Shoshitaishvili; Antonio Bianchi; Aravind Machiry; John Grosen; Paul Grosen; Christopher Kruegel; Giovanni Vigna
DREAM is a decompiler whose control-flow structuring recovers loops, conditionals and switch statements from a recovered control-flow graph without matching predefined region patterns. For acyclic regions it computes each node's reaching condition, then refines the resulting conditions into if-then-else and switch constructs; cyclic regions are structured as loops whose type and continuation condition are inferred. Semantics-preserving transformations first rewrite cyclic regions with multiple entries or successors into single-entry, single-successor regions, and a Boolean variable keeps a repeated test equivalent when its operands may change. On GNU coreutils the authors report no goto statements, against thousands from Hex-Rays and Phoenix, and more compact code than Hex-Rays for 72.7% of functions and than Phoenix for 98.8% in one-to-one comparisons.
Correctness was measured on source-level control-flow graphs: the authors extracted CFGs of the preprocessed coreutils 8.22 source with joern, restructured the 1,530 functions joern parsed (208 of 1,738 failed to parse), recompiled and passed the coreutils tests. That experiment checks the structuring algorithm, not end-to-end decompilation of binaries, variable or type recovery. Structuredness and compactness were measured on the coreutils 8.17 binaries from the Phoenix evaluation, using functions decompiled by all three tools, with line counts as the compactness metric. Phoenix was unreleased, so its output came from its authors, and the 2014 Hex-Rays v2.0.0.140605 baseline is long superseded. The malware case studies cover one sample each from ZeusP2P, SpyEye and Cridex and compare against Hex-Rays only. The authors leave open whether more compact code is always easier to understand. The paper links no implementation, and these results do not describe current decompilers.
Authors & citation
Khaled Yakdan; Sebastian Eschweiler; Elmar Gerhards-Padilla; Matthew Smith
Trains visual-code models using separate code and rendered-output rubrics for data filtering and reinforcement learning. Relevant evaluations include rebuilding webpages, charts and vector graphics from images.
Combines visual reconstruction with text-driven generation. Judge scores do not establish behavioral equivalence. Training and evaluation share some judge families; the human check uses 100 samples. Instruction n-gram filtering cannot exclude all pretraining exposure.
VisStateCoder links visual prototype elements, project files, interaction requirements and render evidence in a project state graph, then uses those links to generate websites and limit subsequent repairs to selected files.
Requires visual prototypes and textual workflow requirements. The five-model Vision2Web evaluation covers up to seven-page HTML/CSS/JavaScript sites with an optional Node.js server; larger projects and React/Vue are untested. Each configuration runs once, without variance or significance estimates. Internal repair checks do not execute interaction workflows; functional scores come from separate benchmark tests. Visual judges and incomplete tests limit conclusions about behavior, maintainability and accessibility.
TaoD2C-Bench reconstructs interfaces from design images, layer metadata and component-library documentation. It separates requirement inference from implementation, checking component choices and layout relations through source/runtime evidence alongside rendered appearance.
Eight models are evaluated on 421 designs, not the full 2,861-design collection. Results include specified retries and one output-repair attempt. The tasks assess component/layout constraints and visual fidelity, not complete application behavior; all designs come from one commercial ecosystem.
Benchmarks website development from visual prototypes and written requirements across 100 static-page, 66 interactive-frontend and 27 full-stack tasks, combining workflow-based functional checks with model-judged visual comparison.
The full-stack tier is specification-driven, not reconstruction solely by exploring a live website. Workflows test state management and CRUD outcomes, but finite agent judgments do not establish exact backend or durable-database equivalence. The current public evaluator maintains a browser session within each workflow and permits local business-API checks; this is not a universal persistence proof.
Benchmarks reconstruction of interactive charts from a reference screenshot, local data and specified actions. Its TRAIL framework uses Developer and Inspector agents with browser interaction traces to diagnose errors and revise the HTML/JavaScript implementation.
Covers 377 single-file HTML/JavaScript tasks with supplied data and explicit interaction requirements. Four models are evaluated with at most one repair round. Failing the Executability gate gives zero on scored dimensions; other rubric judgments use an MLLM, with human validation on 200 Direct outputs from 100 tasks and two models. Finite interactions and judge agreement do not establish complete application equivalence or recovery of hidden data and unspecified behavior.
Uses intermediate Webcrack rewrites to supervise Coconut-style latent reasoning for JavaScript deobfuscation. Training exposes staged code transformations; inference receives obfuscated code and generates a cleaned program in one shot.
A work-in-progress study of one 3B backbone: 100 programs sampled from the 300 smallest benchmark inputs, split 60/20/20. Of 20 test outputs, 10 parsed and 8 matched teacher-derived outputs; the reported 80% is conditional on valid syntax. Semantic checks include manual inspection; stronger baselines and stage ablations remain future work.
Evaluates recovery of web-app behavior from a live reference with hidden source. It provides 4,063 feature-restoration tasks across 26 applications and a separate 12-application reconstruction setting, scored by replaying dependent browser workflows.
Tests cover finite, model-mined behaviors in self-contained apps, including SaaS clones. Persistence failures can be detected through replayed workflows, but passing does not establish all backend or database invariants. Evaluated agents receive structured browser observations without screenshots; visual fidelity is not measured. Public verifiers and reference solutions are not a contamination-resistant test set.
Authors & citation
Jeonghye Kim; Minseon Kim; Young Jin Kim; Matheus Pereira; Marc-Alexandre Côté; Alessandro Sordoni; Xingdi Yuan; Zhengyan Shi
Uses typed browser interaction contracts to check progress and preserve previously verified capabilities while improving generated web code. Only accepted transitions become training examples.
The study concerns web-code generation and self-improvement, not recovery of original source code. Acceptance covers declared interactions and preserved capabilities, not equivalence to an arbitrary original application. Distinct from the other WebWorld web-simulator paper.
WebCompat evaluates screenshot-reconstructed webpages across browsers and devices. Its XCompat detector combines rendered DOM geometry, viewport metadata and screenshots to identify layout and element failures, independently of similarity to the original design.
Filtering retains 254 static pages from 480 generations, producing 2,032 comparison pairs. The held-out detector test uses 51 pages/408 pairs. Failed renders and interaction-dependent layouts are excluded; heuristic errors and subjective design intent limit generalization to dynamic sites or every environment.
UI2App evaluates runnable React/TypeScript applications generated from 600 screenshots across 95 web applications. It separates build executability, visual fidelity and screenshot-implied interaction behavior, including cross-route state, using a reference interaction taxonomy and public evaluation pipeline.
Static screenshots underdetermine behavior; the rubric accepts multiple valid implementations rather than proving equivalence to the original application or backend. Auto IIS judges generated source and screenshots without tool execution; human runtime validation covers a 45-application subset. Cross-route state is not necessarily durable persistence across reloads or sessions. Main results use one generation per application-model pair, with repeated runs on a subset.
Evaluates agents that actively explore a running dashboard and reconstruct its appearance and interaction logic. DashboardMimic provides 180 Plotly+Dash examples, with code-semantic analysis and dynamic interaction tests for the generated implementations.
Limited to Plotly+Dash dashboards, including model-augmented examples. Some settings expose DOM information and evaluation uses model judges. This is not a general backend-recovery benchmark. The linked dataset requires accepting access conditions and sharing contact information.
UniCoder trains a visual-to-code model for charts, SVGs and webpages. Its reinforcement-learning objective combines rendering checks, image similarity and code-derived visual attributes; low-reward training groups receive reference-code trajectories to improve exploration.
Experiments concern static visual reconstruction with one 8B backbone, including Design2Code and ScreenBench. Reference code is used during training; successful rendering does not prove interactive behavior. Extending the attribute extractor to new domains requires new schemas/prompts. Interactive interfaces, animation and mobile-app behavior are not evaluated.
Uses Gemini to locate the prelude functions that Obfuscator.IO inserts for string obfuscation, then applies constant-propagation and inlining passes in the JSIR compiler IR to restore string literals and API names deterministically.
The method addresses Obfuscator.IO string obfuscation; control-flow flattening, other obfuscators and an agent-based design are future work. Evaluation obfuscates 3,000 programs from the ETH 150k JavaScript dataset with four Obfuscator.IO presets and measures prelude detection, recovered literals and run time, not functional equivalence. Production deployment and scan volume are author-reported, and production pre-filtering uses a proprietary YARA rule.
Splits screenshot-to-code generation among grounding, planning and generation agents that label UI regions, build a hierarchical layout tree and write HTML/CSS. The pipeline also produces the Screen-10K training pairs used for supervised and reinforcement fine-tuning, and the paper introduces the 1,000-page ScreenBench.
Output is static HTML/CSS layout; interaction, JavaScript behavior and backend logic are not reconstructed. Results are visual-similarity scores (CLIP and OCR-matched text blocks) on ScreenBench and Design2Code, which do not establish code-level or functional equivalence. The released pipeline calls a hosted model API, Doubao by default. The reviewed text is arXiv v2; the accepted EMNLP version may differ.
Authors & citation
Yilei Jiang; Yaozhi Zheng; Yuxuan Wan; Jiaming Han; Qunzhong Wang; Michael R. Lyu; Xiangyu Yue
Tests reconstruction of interactive webpages from user-interaction videos and supplied static assets. Its 113 tasks replay annotated action sequences to assess visual fidelity and functional behavior across observed page states.
The required output is one HTML file; the paper explicitly instructs models to mock implied backend logic with static JavaScript data. Trajectory coverage and model-based judgments do not prove complete website or backend equivalence.
Authors & citation
Yang Chen; Minghao Liu; Yufan Shen; Yunwen Li; Tianyuan Huang; Xinyu Fang; Tianyu Zheng; Wenxuan Huang; Cheng Yang; Daocheng Fu; Jianbiao Mei; Rong Wu; Licheng Wen; Xuemeng Yang; Song Mao; Qunshu Lin; Zhi Yu; Yongliang Shen; Yu Qiao; Botian Shi
JSimplifier recovers readable JavaScript from obfuscated scripts in three stages. A preprocessor normalizes malformed or mixed-encoding input; a deobfuscator combines AST-based static simplification with selective execution of obfuscated call sites in a Node.js vm context, feeding the results back into the AST; and an optional humanizer renames identifiers with an LLM, adapting HumanifyJS, then formats the output with Prettier. The authors catalog 20 obfuscation techniques in four categories and assemble MalJS (23,212 malicious scripts) and BenignJS (21,209 scripts). They report that JSimplifier processes all 44,421 samples and reduces Halstead length by 88.2% on 1,296 multi-technique JsDeObsBench samples.
The headline “100% correctness” is syntactic validity on 400 samples (100 BenignJS, 300 MalJS) checked by parser and by hand; behavioral consistency, judged with Joern CFG/DDG comparison and manual review, holds for 83 of 100 samples. The 20-technique capability test applies each technique to one representative snippet, and the static and dynamic stages alone handle 19; identifier renaming needs the LLM stage. Readability gains come from four LLMs (Claude 3.7 Sonnet, Gemini 2.5 Pro, DeepSeek-R1 and GPT-o3) scoring 60 MalJS pairs on a 0–10 scale; a separate check of GPT-4o-mini renaming on 100 samples found 12% hallucinated names, and the user study had 9 participants. The entropy comparison excludes the LLM stage. MalJS was collected with a security company and BenignJS mixes Tranco top-site scripts with 2,000 clean samples obfuscated by the authors. The authors exclude cryptographic packing, custom interpreters and time-dependent decryption, and note that ES6+ syntax may be downgraded and that vm-based execution carries escape risk. The output is readable JavaScript, not a reconstructed website.
Authors & citation
Dongchao Zhou; Lingyun Ying; Huajun Chai; Dongbin Wang
Builds an execution-verifiable JavaScript deobfuscation benchmark by applying seven JavaScript-Obfuscator transformations to Project CodeNet programs with test cases, then scores LLM and tool outputs for syntax, execution, code simplification and similarity to the original code.
On average, LLM outputs passed execution checks 60.93% of the time, against 83% for the two existing deobfuscator baselines. All samples come from a single obfuscator applied to moderate-length competitive-programming solutions. The 4,515 obfuscated malware samples have no test cases and are compared only through behavior traces.
Benchmarks reconstruction of interactive webpages from screenshots before and after an interaction. The final study covers 127 webpages and 374 interactions, comparing rendered appearance, automated interaction checks, and human usability judgments.
The inputs specify individual interaction transitions, not autonomous exploration of a complete site. Its tests do not establish long-workflow, server-side, or original-source equivalence; a detectable interaction is not necessarily usable.
Authors & citation
Jingyu Xiao; Yuxuan Wan; Yintong Huo; Zixin Wang; Xinyi Xu; Wenxuan Wang; Zhiyao Xu; Yuhang Wang; Michael R. Lyu
LaTCoder converts webpage design images to HTML/CSS by making layout an explicit intermediate step. A rule-based divider recursively finds solid-colored horizontal or vertical lines that avoid OCR-detected text, then merges small regions into grid-aligned blocks. An MLLM writes code for each block with a chain-of-thought prompt. The blocks are then assembled by absolute positioning or by the MLLM, and a reference-free verifier combining MAE and CLIP picks the better result. The paper also introduces CC-HARD, 128 complex-layout pages manually sampled from Common Crawl.
Targets static HTML/CSS from a design image; JavaScript behavior, interactions, navigation and backends are outside the method, and it does not recover a site's original source. Evaluation uses Design2Code-HARD (80 pages) and CC-HARD (128 pages), scoring generated code and screenshots against references with TreeBLEU, CLIP, Visual Score and MAE. Reported gains are relative changes on small absolute scores: with DeepSeek-VL2 on CC-HARD, TreeBLEU rises from 0.09 to 0.15 (+66.67%), and CLIP falls slightly on Design2Code-HARD. The divider assumes rectangular, solid-colored separators and fixed pixel thresholds; the block-merge area was tuned on CC-HARD. Short-context models use absolute positioning only, which fixes block positions from the design. Block content can still be misplaced, and Gemini sometimes omitted blocks during assembly. Pairwise human preference (at least 60% for LaTCoder) covers GPT-4o on CC-HARD only, by majority vote of six annotators. GPT-4o, Gemini and DeepSeek-VL2 results are historical baselines, not current rankings.
Authors & citation
Yi Gui; Zhen Li; Zhongyi Zhang; Guohao Wang; Tianpeng Lv; Gaoyang Jiang; Yi Liu; Dongping Chen; Yao Wan; Hongyu Zhang; Wenbin Jiang; Xuanhua Shi; Hai Jin
Diagnoses webpage-to-code capability through visual perception, HTML programming, image-to-HTML understanding, and screenshot reconstruction tasks. It separates underlying skills using question-answer pairs derived from real websites.
Primarily evaluates visual and HTML reconstruction skills. The paper includes a preliminary interaction study, not a full workflow or backend-equivalence benchmark. The official repository currently exposes README and images but not the evaluation scripts described in its instructions.
WAFFLE fine-tunes multimodal LLMs to generate HTML/CSS from UI screenshots. A quarter of the decoder's attention heads use a structure-aware mask that limits each element's tokens to its parent, preceding siblings and its own earlier tokens. A contrastive objective aligns image and code embeddings over groups of four mutated WebSight-v0.1 pages. With VLM-WebSight and Moondream2 backbones, the authors report up to 9.00 pp higher HTML-Match and 0.0982 higher CW-SSIM than standard fine-tuning on WebSight-Test, plus gains on every Design2Code metric.
Generates static HTML/CSS from a single screenshot; it does not recover a site's original source, scripts, interactions, navigation or backend. HTML-Match strips styles and attributes before requiring pixel-identical renders, while CW-SSIM, CLIP and LLEM are similarity scores; the authors note that these metrics do not fully capture human judgment. Training data and the 500-page WebSight-Test are synthetic WebSight-style pages; Design2Code supplies 484 real-world pages. Only two backbones were fine-tuned, and on Design2Code the Moondream2 model trained with WAFFLE stays below GPT-4o and GPT-4o mini. The Gemini 1.5 Pro baseline answered 384 Design2Code instances. Human ranking used 50 samples per dataset and two annotators. Commercial-model prompting results are historical baselines, not current rankings.
2025Proceedings of the ACM on Software Engineering (FSE 2025)Journal article
Recursively divides a webpage screenshot along visual separation lines, has a multimodal model generate HTML/CSS for each leaf segment, and assembles child code upward into the full page; evaluated with GPT-4o, Gemini and Claude models and a small developer study.
It reconstructs static visual layout as HTML/CSS, not interaction, JavaScript behavior or backend logic. Test pages are self-contained snapshots with images replaced by placeholders; the main evaluation samples 111 of 348 top-ranked sites, adds 80 harder GitHub Pages pages, and the usefulness study has four participants. Segmentation uses manually set thresholds, and assembly can still misarrange elements.
Authors & citation
Yuxuan Wan; Chaozheng Wang; Yi Dong; Wenxuan Wang; Shuqing Li; Yintong Huo; Michael Lyu
Evaluates generation and conversion of structured outputs, including HTML, React and SVG, with separate checks for format adherence and structural correctness.
Not a software reverse-engineering or GUI-navigation benchmark. Structured-output scores do not prove recovered source identity, backend reconstruction or behavioral equivalence; aggregate scores are not fully correct-output percentages.
WebCode2M pairs about 2.56 million real-world webpage screenshots with purified HTML/CSS code and element bounding-box layout trees. Pages sampled from a Common Crawl snapshot are cleaned (scripts, metadata and hidden elements removed, styles merged into one block), rendered with Playwright, and filtered by a neural scorer trained on 10,000 annotated samples. The paper fine-tunes Pix2Struct-1.3B as the WebCoder baseline, introduces TreeBLEU to measure DOM-structure recall, and benchmarks specialized, open and commercial multimodal models on short, medium and long test subsets of 256 pages each.
The task is static HTML/CSS generation from a design image. Scripts are stripped during purification, so interaction, client-side behavior and backend logic are not reconstructed, and outputs are judged by visual score, CLIP similarity and TreeBLEU rather than functional equivalence. The paper reports that no evaluated model reproduces image elements accurately, and scores fall as target code grows longer. The 768 test pages were drawn from high-scoring candidates and manually inspected; the scorer is about 90% accurate, so low-quality training samples remain, and the authors warn that crawled content may still include inappropriate material. Commercial models are named only by family, and about one-third of Gemini test cases failed with recitation errors, so those comparisons are snapshots. The dataset card's CC BY 4.0 license does not clear rights in the crawled pages' code, text or images.
Authors & citation
Yi Gui; Zhen Li; Yao Wan; Yemin Shi; Hongyu Zhang; Bohua Chen; Yi Su; Dongping Chen; Siyuan Wu; Xing Zhou; Wenbin Jiang; Hai Jin; Xiangliang Zhang
Sketch2Code benchmarks vision-language models on turning rough hand-drawn wireframe sketches into HTML webpage prototypes. Crowd annotators drew 731 sketches of 484 real-world webpages from the Design2Code test set. Ten models, eight commercial and two 8B open-source, generate pages directly and in two multi-turn settings with a GPT-4o simulated user: following its feedback, or asking it clarification questions. Outputs are scored by Design2Code visual similarity and a new IoU-based layout similarity over component types such as text blocks, images, navigation bars, forms and buttons. Claude 3.5 Sonnet and GPT-4o lead single-turn layout similarity, and feedback rounds help commercial models more than question asking does.
The generated pages are prototypes scored by visual and layout similarity to reference screenshots, not recovered source code; behavior, interactivity and code equivalence are not evaluated, and page images are replaced with a placeholder. Sketches deliberately omit most text and styling, so similarity scores are low in absolute terms, with best single-turn layout similarity around 22%. The simulated user is GPT-4o, judged faithful in 83.3% of feedback and accurate in 86.7% of answers by human annotators. Only 8B open-source models were run, and a multi-turn example needs 40,000 to 160,000 input tokens. The user study interviewed eight UI/UX experts. Results describe 2024 model versions.
Evaluates reconstruction of frontend code from webpage screenshots using real-world reference pages, visual-element and layout metrics, and human assessment.
This is visual website reconstruction, not binary decompilation or recovery of backend behavior and interactive workflows. The main benchmark contains 484 webpages with placeholder images; code and data have separate licensing terms.
Authors & citation
Chenglei Si; Yanzhe Zhang; Ryan Li; Zhengyuan Yang; Ruibo Liu; Diyi Yang
Evaluates GPT-3.5-turbo, GPT-4, LLaMA-2-13B, Code-LLaMA-2-13B-Instruct and StarChat-Beta on explaining what de-commented JavaScript, Python and C code does, then repeats the task on JavaScript obfuscated with JavaScript Obfuscator (identifier renaming, dead-code injection, control-flow flattening and string splitting) and with Wobfuscator’s WebAssembly transformation. GPT-3.5 and GPT-4 are also asked to rewrite 100 IOCCC winning programs as readable, compilable code. Case studies apply GPT models to a botnet and a benign browser-extension repository, an Android SMS stealer decompiled with JD-GUI, and WannaCry code decompiled with RetDec.
Ground truth is GPT-4’s own explanations after manual checking by four students; other models are scored against those texts with similarity metrics and a GPT-4 judge, which the authors say is unreliable without manual calibration. Correctness is judged at explanation level, not by recovered behavior or executable equivalence. Only JavaScript is obfuscated with generated transformations; neither GPT model deciphered the WebAssembly that Wobfuscator inserts, and the smaller models produced no meaningful explanations once obfuscation was applied. Author-reported figures include 97.4% GPT-4 accuracy on non-obfuscated code and 87% on obfuscated JavaScript; in IOCCC de-obfuscation, about 20% of GPT-3.5’s targets and 19% of GPT-4’s yielded compilable code, and only 8% and 4% produced correct results. The malware analyses are qualitative case studies on a few samples, with manual chunking for WannaCry, missed behavior when context was split, and a false alarm in the botnet repository. Models recognized code from popular repositories, so memorization is a confound. The evaluated GPT-3.5/GPT-4-era and LLaMA-2-13B-class models are dated. The study analyzes source or already-decompiled text; it does not perform binary decompilation.
Authors & citation
Chongzhou Fang; Ning Miao; Shaurya Srivastav; Jialin Liu; Ruoyu Zhang; Ruijie Fang; Asmita; Ryan Tsang; Najmeh Nazari; Han Wang; Houman Homayoun
WebSight is a synthetic dataset of 2 million HTML documents paired with full-page screenshots for training screenshot-to-HTML models. Mistral-7B-Instruct generates website concepts, Deepseek-Coder-33B-Instruct writes each page in HTML with Tailwind CSS, images come from keyword-based stock-photo URLs, and Playwright renders the screenshots after filtering. This v0.2 release replaces the 823K-pair v0.1, which used traditional CSS and colored rectangles instead of images. The authors fine-tune an 8B vision-language model, Sightseer, on the data with DoRA and show qualitative conversions, including one from a hand-drawn sketch.
The websites are synthetic, LLM-generated single pages, not crawled real sites, so training on them does not establish reconstruction of real websites, their source or their behavior. Evaluation is qualitative only: no benchmark scores are reported, checkpoints were selected by manual inspection because validation loss did not track output quality, and the authors note failures on complex layouts, long text and designs unlike the training data, as well as Tailwind syntax errors. v0.2 pages load Tailwind CSS from a CDN link and images from source.unsplash.com URLs, so re-rendering depends on those external services. The base model was unreleased at the time, and the report is not peer-reviewed.
2024NeurIPS 2024 Datasets and BenchmarksConference paper
Studies screenshot-to-HTML reconstruction through instruction tuning and two evaluation benchmarks. Web2Code combines generated webpages, refined WebSight and pix2code image/code pairs, and WebSRC-derived question answering. Models produce HTML from a webpage image and instruction; rendered output is compared visually, while a separate benchmark measures screenshot understanding.
Evaluation uses 1,198 screenshots and 5,990 yes/no questions. WCGB grades rendered images with GPT-4V; its User Interface and Interactivity category measures element appearance, not executed click paths, persistent state or backend behavior. Synthetic and refined examples do not establish coverage of arbitrary real web applications, original-source equivalence or code quality. The paper reports contact-information issues in 2.7% of a 1,000-item privacy sample, so a blanket claim that the data is free of such information is unsupported.
Clematis helps developers understand a JavaScript web application's runtime behavior from a recorded user session. A proxy instruments JavaScript on the fly to log DOM events, timeouts, XMLHttpRequest phases, function calls and DOM mutations. The trace is grouped into episodes, each with its triggering event, executed functions and resulting DOM changes, linked by temporal and causal edges and shown in an interactive visualization. In an industrial study with 20 developers on the Phormer photo gallery, the authors report 61% higher task accuracy and 47% less completion time than developers using their preferred tools.
A dynamic program-comprehension aid for client-side behavior exercised in one recorded session; it does not reconstruct server-side logic, generate a replacement application or establish behavioral equivalence. Code run through eval is not captured. Evaluation consists of two small controlled experiments on comprehension tasks: 16 UBC participants on a lab-built survey application against Firebug, and 20 professional developers on Phormer. The authors graded accuracy against a prewritten answer key. Overhead was measured only for one minute of Phormer activity. The 2014 prototype drives Firefox through Selenium and a WebScarab proxy; it is not shown to run on current browsers, and the comparison tools are historical.
Authors & citation
Saba Alimadadi; Sheldon Sequeira; Ali Mesbah; Karthik Pattabiraman
Crawljax infers a state-flow graph of an Ajax web application by driving it in a real browser. It scans the DOM for candidate clickable elements, fires events on them, compares the resulting DOM against known states with configurable comparison and similarity thresholds, and records new states and the event transitions between them, backtracking by reloading and replaying paths. A concurrent variant partitions the crawl across browser instances. The Java tool uses Selenium WebDriver and plugins; one plugin writes a static mirror of the crawled states. The authors evaluate six open-source and industrial sites and measure how the number of concurrent browsers affects crawling runtime on Google AdSense.
Crawljax recovers client-side UI states and navigation transitions, not application source code, server-side logic or a complete behavioral model. The inferred graph is one snapshot instance: click order, server-side factors and state-explosion limits such as depth, time and state caps affect what is found. With default settings it clicks each element once and missed elements that only appear after repeated clicks: 33 of 36 annotated elements were found in Pet Store, and 38 of 40 sampled clickables across four external sites. No gold standard or comparable Ajax crawler was available, so accuracy was assessed by injected elements, a manual reference model and spot checks. External sites change over time, and the concurrent study ran at Google on AdSense. The reviewed 2023 repository is a later Java 11 and Selenium 4 codebase, not the 2012 evaluation build.
Infers executable models of game state and transition rules from interaction. Agents revise these models against recorded history, search within them, and compare predicted observations with real outcomes while executing plans.
Evaluation uses 25 public ARC-AGI-3 games, 21 public DiG-bench games and MazeBench. Matching recorded transitions establishes history consistency, not original-source recovery or correctness on unseen states. RHAE measures completion and environment-action efficiency, not elapsed time or total computation. Component ablations cover ten ARC games; MazeBench compares recorded trajectories with a leaderboard median rather than a controlled human experiment. Transfer to arbitrary software or stochastic environments is not demonstrated.
The 247 tasks derive from 41 games. Inputs vary by task; porting receives Godot source, so this is not purely black-box reconstruction. Each model uses one framework, confounding their effects. Finite scenario checks and 100-game functional validation do not establish general gameplay equivalence.
Evaluates whether vision-language agents can infer hidden game rules through screenshots and active interaction, then build runnable single-file HTML replicas across 125 tasks from 25 web games, each with five difficulty levels.
The tasks cover rule-driven web puzzles and board games, not native or commercial game reconstruction. Rule, implementation, and visual scores use LLM judges and do not prove behavioral equivalence. The official site provides demos; a benchmark-code repository was not verified.
Tycho constructs editable Python models of game state, transitions, rendering and outcomes from action/observation histories in unfamiliar ARC-AGI-3 games. It checks model predictions against recorded transitions and can use, repair or bypass these models during planning.
Evaluation uses one stochastic run per configuration on 25 public games and 183 levels; five games informed harness development. Models may abstain on cells or accept bounded rendering variants. Transition replay and game scores do not establish complete engine reconstruction, correct objectives, unseen-state fidelity, unseen-game generalization or run-to-run reliability; the agent can bypass modeling.
Tests whether agents can recover executable decision policies from behavioral traces and self-designed opponent probes, using 75 synthetic policies across five game arenas and measuring held-out action agreement and downstream competitive value.
The targets are fixed synthetic policies, not full games or adaptive adversaries. Agreement on finitely evaluated states does not uniquely identify source code, and some arenas have substantial variance. The public package omits some paper-only plotting and ablation suites; reproduction also needs CodeClash, a container runtime, and model access.
Authors & citation
Babak Rahmani; Sebastian Dziadzio; Joschka Strüber; Sergio Hernández-Gutiérrez; Matthias Bethge
2026CHI EA 2026 (poster/extended abstract)Conference paper
Transforms game-interface screenshots into editable engine assets through hierarchical YAML, visual grounding, asset extraction, and Unity UXML/USS synthesis, with evaluation on a curated game-UI benchmark and review by three senior UI/UX designers.
The generated interactions cover basic hover, toggle, and click behavior. Complex temporal and drag-and-drop logic remain outside scope, and translucent overlap or ambiguous diegetic layouts can fail. This is game-UI prototyping, not complete gameplay reconstruction; the official page still labels code as coming soon.
Authors & citation
Yunshu Bai; RuiHao Li; Hao Zhang; Chien Her Lim; Ming Yan; Mengtian Li
PoE-World represents an Atari world model as an exponentially weighted product of many small Python programs, or programmatic experts, each synthesized by an LLM to explain one aspect of the observed dynamics; learned weights and optional synthesized hard constraints combine them. From object lists that OCAtari parses out of a demonstration of fewer than 1,000 frames, the authors learn stochastic, partially observed models of Pong and Montezuma's Revenge and use them in a planning agent, including on rearranged Pong-Alt and Montezuma's Revenge-Alt variants that have no demonstrations.
Models consume symbolic object lists, not pixels, and OCAtari had to be patched manually for each game, so evaluation covers only Pong and Montezuma's Revenge plus two author-built variants. The learned programs approximate observed dynamics and are not the games' original code: next-observation prediction accuracy on 1,000 random test frames is 0.06 for Pong and 0.31 for Montezuma's Revenge, and object-attribute accuracy is 0.66 and 0.76. Synthesized experts build on manually written helper classes and contact functions and were generated with gpt-4o-2024-08-06. Exploration and reward learning are outside the method; agent scores use 5 seeds and at most 3k training steps after the demonstration, and the planner reached the key in 5 of 9 Montezuma's Revenge tries.
Authors & citation
Wasu Top Piriyakulkij; Yichao Liang; Hao Tang; Adrian Weller; Marta Kryven; Kevin Ellis
GameNGen learns to simulate DOOM interactively with a diffusion model instead of game code. A PPO agent first plays the game in ViZDoom, and its recorded trajectories, from random to trained play, become the training data. Stable Diffusion v1.4 is then fine-tuned to predict the next frame from the last 64 frames and actions, with noise added to context frames to keep long autoregressive rollouts stable and a fine-tuned latent decoder for detail. With 4 sampling steps it runs at 20 frames per second on one TPU-v5. The authors report next-frame PSNR of 29.43 and human raters picking the real game in 58–60% of short clips.
A neural approximation trained on recorded gameplay of an existing game; it recovers no game logic, source code, assets or state representation and is not an exact simulation. The model sees just over 3 seconds of history, so state outside that window survives only through on-screen cues and learned heuristics, which can be wrong; for example, repeated shooting may spawn enemies. Locations and interactions the agent never explored are simulated poorly. Training used 128 TPU-v5e devices and 70M agent frames at 320×240, plus an environment-specific reward function. PSNR is a teacher-forced single-frame measure on 2,048 held-out trajectories from 5 levels; autoregressive rollouts diverge from ground truth after a few steps. Human evaluation used 10 raters on 1.6- and 3.2-second clips and 150 later 3-second clips; the authors say they could often spot the simulation within seconds. Only DOOM was tested, and the model cannot readily create new games.
WorldCoder learns executable Python transition and reward models from environment interactions and natural-language goals. It refines code against recorded experiences and an optimistic planning constraint, studying transfer and task solving in Sokoban, MiniGrid and ALFWorld.
Assumes deterministic dynamics and symbolic, discrete states rather than recovering a game from arbitrary pixels or binaries. Consistency with observed transitions and successful planning do not establish complete engine equivalence or correctness on every unseen state. Evaluation concerns bounded environment models and planning, not original source or asset recovery.
GIF-MCTS generates executable Python transition and reward models from natural-language environment descriptions and offline state-transition trajectories, using prediction feedback to generate, improve and repair code. CWMB evaluates 18 reinforcement-learning environments and downstream planning; a separate experiment uses a simplified Read to Fight Monsters grid world.
Assumes deterministic, fully observable dynamics and sufficiently detailed descriptions; this is not recovery from pixels or binaries alone. Accuracy uses the replay transitions supplied during synthesis, alongside planning-return evaluation, rather than proving correctness on all unseen states or full-engine equivalence. RTFM fixes the manual across episodes and keeps monsters stationary. CWMB also includes non-game control and physics tasks; some planning returns are unavailable and complex continuous-state models remain difficult.
Authors & citation
Nicola Dainese; Matteo Merler; Minttu Alakuijala; Pekka Marttinen
2023Proceedings of the ACM on Programming Languages (POPL 2023)Journal article
AutumnSynth synthesizes a program in Autumn, a functional reactive language for Atari-style grid worlds, from a trace of observed frames and user actions such as clicks and arrow keys. Heuristic perception parses each frame into typed objects and tracking links them across time steps. The synthesizer then finds an update function for each observed change and searches for the event that triggers it. When no event over the observed state explains an update, automata synthesis invents latent state, such as counters or modes, using a SAT-based Sketch encoding or a faster heuristic. The authors introduce the 30-program Causal Inductive Synthesis Corpus (CISC), on which the heuristic variant synthesizes 27 programs. A slightly modified version synthesizes 21 of 27 games from the third-party EMPA grid-world corpus.
Targets are small, fully observed grid worlds expressible in Autumn, not games with continuous physics, rich graphics or compiled code. Perception and tracking are heuristic, and because Autumn renders objects partially transparent, perception does not address occlusion. CISC input traces were constructed manually to exercise each model’s dynamics, and success means the synthesized program reproduces the given trace, not semantic equivalence to the ground truth. Heuristic runtimes range from about two minutes to 9–13 hours; the Sketch variants time out after 24 hours on several benchmarks, and EMPA runs time out after 40 hours on some games. EMPA required domain-specific changes for random events. Because most EMPA games are nondeterministic, their results were checked by manual inspection, and only the first level of each game was used. On new CISC test traces, four synthesized programs diverged from the ground truth because their training traces did not show all dynamics, and concatenating several traces recovered the ground-truth program for the three that were retested. For random events AutumnSynth detects nondeterminism but does not infer the underlying probability distributions. The method is symbolic program synthesis and does not use an LLM.
Authors & citation
Ria Das; Joshua B. Tenenbaum; Armando Solar-Lezama; Zenna Tavares
MappyLand, a Rust rewrite of the earlier Mappy system, instruments the FCEUMM NES emulator core to map action/adventure games as they are played from controller inputs. It hooks accesses to the picture-processing unit's scrolling registers and reads emulator memory to register scrolling, separate the playfield from menus, detect and track sprites, build grid-based tile maps and recognize room transitions. An online merging step combines observed rooms into a consistent room graph across backtracking and multiple play sessions. In a 34,514-frame test the authors report about 1.5 ms of mapping overhead per frame, against about 0.2 ms for emulation alone.
Limited to NES action/adventure games in which the player controls characters moving through a 2D, tile-based space with a single physical layer; independently scrolling layers and parallax backgrounds are unsupported, and other consoles are left to future work. The method relies on emulator instrumentation and memory reads rather than pixels alone, and needs the game image plus controller inputs. Maps were judged accurate only by human inspection of the authors' play on a sample of games; with no reference dataset, no automated quantitative accuracy evaluation was performed. Areas not seen during play stay unmapped, and visually similar rooms can be ambiguous. Timing comes from individual reported runs; the authors expect, but do not measure, similar overhead across NES games. Output is maps, room graphs and entity tracks, not game code, rules or assets.
Authors & citation
Joseph C. Osborn; Adam Summerville; Nathan Dailey; Soksamnang Lim
GameGAN learns an action-conditioned neural simulator of a game from recorded frames and the actions taken, without access to the game’s logic or engine. An action-conditioned LSTM dynamics engine with a stochastic latent variable predicts the next state, an optional external memory module builds an internal map so that revisited locations stay consistent, and a rendering engine decodes frames while separating static from dynamic components. Training is end-to-end with adversarial losses, including an action-conditioned discriminator, and a temporal cycle loss for memory consistency. The authors evaluate on a modified Pac-Man, a ghost-free Pac-Man-Maze and VizDoom “take cover”, training RL agents inside the learned simulators and testing them in the real games, and measure long-term consistency with a come-back-home task; a human-playable model of the official Pac-Man is shown qualitatively.
GameGAN imitates a game’s pixels and dynamics; it does not recover game code, symbolic rules, assets or an executable. Training needs large logged datasets with actions: 40K Pac-Man episodes from a high-entropy DQN agent that sees the full environment, and 10K random-policy VizDoom episodes, at 84×84 and 64×64 resolution. Quantitative results cover small grid-based Pac-Man variants and one VizDoom scenario, with an external classifier supplying game outcomes. Agents trained in the simulators can exploit modeling errors: the full model sometimes let Pac-Man pass through walls, and its Pac-Man agents scored below those of the memoryless variant, while on VizDoom the World Model baseline scored higher than GameGAN. The released code supports VizDoom training only, relies on a deprecated WorldModels data setup, and is under the non-commercial NVIDIA Source Code License; Pac-Man data and the official Pac-Man model are not included.
Infers platform-game jump mechanics by running controlled experiments in a modified FCEUX NES emulator. A person first plays each game to a safe state; the system then repeatedly holds the jump button for 1 to 120 frames, waits 120 frames and resets, recording every sprite from the PPU sprite table. Sprites that usually touch one another are merged into characters and tracked. A hand-designed four-state jump automaton (ground, rising with and without the button, falling) is then fitted with linear support vector regression, giving per-state velocity and gravity parameters plus minimum and maximum button-hold durations. Across 48 games (about 30% of NES platformers) and 52 characters, the authors cluster the learned jumps into three styles.
Models only vertical jumps from a standing position; horizontal movement, running jumps, double jumps and interactions with enemies or terrain are ignored, and no game code, assets or complete mechanics are recovered. The fixed automaton is assumed rather than learned, so non-physical jumps such as Castlevania's fit with systematic errors. Stair-step arcs, delayed jump starts, unusual landings and animation jitter are acknowledged problems. Each game needs human play to reach a safe state, prior knowledge that it has jumping, and a known jump button. The 48 games are not an exhaustive sample, and results are descriptive comparisons and clusters rather than tests against developer-specified parameters.
Authors & citation
Adam Summerville; Joseph C. Osborn; Christoffer Holmgård; Daniel W. Zhang
Learns a rule-based forward simulator of a 2D game from gameplay video. Using a supplied sprite palette, OpenCV locates sprites in each frame; matched sprites are converted into hand-designed fact types for animation, position, relative distance, x and y velocity and camera offset. Game Engine Search then scans the frames, and whenever the current engine’s predicted next frame differs from the real one by more than a threshold, it runs a greedy search over engines that add or modify condition–effect rules or mark them as player-controlled. Engines are ranked by pixel error and rule count. Trained on one Super Mario Bros. Level 1-1 video, the learned engine predicts frames of a different player’s video more accurately than a CNN frame predictor and a previous-frame baseline, and a value-iteration agent planning with it shows no significant difference from one using the Infinite Mario engine’s native forward simulation.
The evaluation covers one level of one game: training uses a single “speedrunner” video of Super Mario Bros. Level 1-1 (1,569 frames), and testing uses an “explorer” video of the same level. Frame parsing depends on a fan-authored sprite sheet and hand-authored fact types, which the authors say makes the method less applicable to more complex domains such as real video. The learned engine is a set of search-derived rules for on-screen sprite behavior, not recovered game code or a binary; it does not represent player death or level transitions, and mechanics absent from Level 1-1 are not expected to transfer. Search can be slow at low thresholds, up to two weeks for one level on a 2013 iMac, and the reported run used a zero threshold. The CNN comparison used downscaled grayscale frames, and the gameplay comparison is indirect, through agents in the Infinite Mario engine.
ReviveBench evaluates repair of nonrunning software and source-unavailable implementation of bounded computational and transactional engines from open specifications, using hidden reference-based checks. It also audits grading defects that can wrongly accept or reject reconstructions.
Studies ten revival and thirteen bounded engine tasks; reconstruction receives public specifications and three worked examples. Most model/task cells have one or two seeds; the reconstruction matrix reports the best eligible run. Excluded infrastructure failures and differing API adapters limit comparisons. Task-level pass thresholds do not require every check to pass, and CFD passes cannot establish numerical-solver capability. Verifier audits and contamination controls do not prove general equivalence or rule out prior exposure.
Uses seven agents and human review to recover business requirements from source, tests, configuration and documentation. A reported enterprise migration checks extracted rules against production-defect records.
Evidence covers one enterprise Java programme and one legacy pilot. The cost comparison uses a modeled manual baseline with asymmetric accounting. Static analysis misses some runtime and external rules; coverage does not establish completeness. Confidential inputs limit independent reproduction.
Studies a RAG-based, human-supervised pipeline for recovering requirements and code traces from two projects. It also tests deliberate generation of unimplemented requirements and requirements smells (potential quality issues), exposing verification and labeling difficulties.
The 188 generated requirements come from a student web application and Mattermost Desktop, using one model and PhD evaluators. Prompt choices, accumulated feedback and author-resolved labels affect validity. Findings concern this small experience study, not all code-to-requirements methods.
Authors & citation
Alexander Korn; Jone Bartel; Max Unterbusch; Andreas Vogelsang
WidgetGen reconstructs widget screenshots as React JSX. OCR text boxes and color palettes guide whole-widget layout reasoning and optional chart generation before one terminal render; generated render/code pairs also support model fine-tuning.
The main comparison uses six models and 1,000 test widgets from one dataset. Its nine image-based metrics do not test interaction, accessibility or maintainability. The supervision experiment covers six Qwen-family checkpoints; broader UI and model-family generalization remains open.
Authors & citation
Houston H. Zhang; Tao Zhang; Li Gu; Linfeng Ye; Yuanhao Yu; Xinxin Zuo; Yang Wang; Zhixiang Chi
Synthesizes executable programs from input/output observations using LLM-guided evolutionary search, progressively revealed examples, simple-to-complex mutations and execution-error feedback. An active variant chooses inputs and queries a hidden reference program.
IO2CodeBench has 125 generated tasks spanning basic operations, algorithms, geometry and image-to-value mappings, normally with eight visible and fifteen held-out examples. Active exploration covers a 25% subset from each level. Multimodal results average sample pass ratios, while other levels count tasks passing every held-out example. Finite test success does not establish original-source recovery or complete equivalence; arbitrary applications and stateful services are not evaluated.
UIOrchestra converts static mobile UI designs into React/CSS through layout-description, code-generation and rendered-image comparison agents. Its APPUI benchmark contains 1,101 human-refined code/rendering pairs based on screenshots from 165 apps across 12 categories. The reference code is reconstructed for evaluation, rather than recovered original application source.
Tests visual, component and code similarity, not native-package execution, interaction, backend behavior or application equivalence. APPUI references are rendered reconstructions of iOS screenshots; unavailable images/icons use placeholders. A human study covers 200 samples with four annotators. The difference-analysis prompt ignores font-size/typeface differences and placeholder image content.
2025Proceedings of the ACM on Software Engineering (FSE 2025)Journal article
Combines visual component extraction, multimodal models, page-transition graphs and iterative checks to reconstruct multi-page mobile UI code from designs or screenshots, evaluated with React Native, Flutter and ArkUI.
People review and correct the inferred navigation graph. Evaluation covers 50 five-page UI sets; transition coverage and compilation do not establish full business-logic or backend recovery. Framework results vary, and visual metrics imperfectly reflect perceived fidelity. Public scripts and data are available, but no project-level license was found at the reviewed commit.
Authors & citation
Ting Zhou; Yanjie Zhao; Xinyi Hou; Xiaoyu Sun; Kai Chen; Haoyu Wang
2025ISSTA Companion 2025 (LLMSC Workshop)Conference paper
CAMA is a framework for benchmarking how code LLMs read decompiled Android malware. Androguard decompiles each APK into Java functions, and each model returns a structured output per function: a summary, a refined function name and a 0–10 maliciousness score. Two analysis tasks build on these outputs. Malicious-function identification is scored by consistency (whether scores and names stay stable when the model is re-prompted with its own summaries) and by fidelity (how much a LightGBM malware-category classifier loses confidence when the top-scored function descriptors are removed). Malware-purpose summarization is scored by semantic relevance (BLEU, METEOR and ROUGE-L against reference app descriptions). The benchmark holds 118 APKs from 6 categories and 13 families, with 7,542,799 decompiled functions. Of the four models evaluated, StarChat-Beta has the highest maliciousness consistency (0.813, against 0.381 for CodeLlama) and narrowly the highest semantic-relevance scores, while fidelity varies widely across models and removal sizes. Renaming functions with model-suggested names raises fidelity for CodeLlama and StarChat but lowers METEOR by about 10%.
This is an eight-page workshop paper that evaluates four open-source code LLMs: CodeLlama-7b-Instruct, StarChat-Beta and the base-size CodeT5 and PLBART, not current frontier models. CodeT5 and PLBART were fine-tuned for three epochs to produce the output fields, and their maliciousness scores come from a separate DeepSeek-R1-Distill-Llama-70B model, so score-based results do not measure those two models alone; both are also excluded from the renaming study. No function-level ground truth exists: malicious-function identification is measured indirectly through classifier confidence, and the reference app descriptions are GPT-4-generated summaries from the earlier LAMD dataset; the authors note that these surrogates can add noise and bias. The 118 samples are de-duplicated LAMD malware decompiled only with Androguard, and obfuscation is not studied separately. Fidelity scores have standard deviations close to their means, for example 0.332 ± 0.30 for CodeT5 at k = 2. The Zenodo record holds the structured model outputs (function summaries, refined names and maliciousness scores); neither it nor the LAMD record states that it contains the APK samples. CAMA does not detect malware and is not a decompiler.
2020IEEE Transactions on Software EngineeringJournal article
ReDraw turns screenshots or design mock-ups into native Android GUI prototypes by detecting components, classifying widget types with a CNN, and assembling a plausible view hierarchy with a data-driven nearest-neighbor method.
The implementation covers 15 Android component classes and one screen at a time; multiple screens require manual combination, and hierarchy construction is tied to the mined screen size. Evaluation concerns appearance and view hierarchy, not recovery of original app logic or backends. Screenshots and reconstructed mock-ups served as proxies for real design artifacts.
Authors & citation
Kevin Moran; Carlos Bernal-Cárdenas; Michael Curcio; Richard Bonett; Denys Poshyvanyk
Evaluation uses synthetic screenshot/code pairs: 1,500 training and 250 test instances per platform. Reported accuracy concerns DSL tokens, not complete apps or behavior. Text labels are ignored; platform datasets train separate models using the same architecture. The task covers layout, components and relationships, not application logic or backend recovery. The repository describes an educational research prototype; its older dependencies were not runtime-tested.
UI2code translates a mobile UI image into an Android component hierarchy and layout skeleton using a neural encoder-decoder. The paper evaluates structural prediction and a small developer study of implementing layouts from generated skeletons.
The output covers component types and hierarchy from a single image; detailed styling and text require developer work. Structural token accuracy and an eight-participant study on five screenshots do not establish complete visual fidelity, navigation, app logic or backend recovery. The main split is screenshot-based; a separate evaluation on 1,208 images from 20 unseen apps provides limited generalization evidence.
Authors & citation
Chunyang Chen; Ting Su; Guozhu Meng; Zhenchang Xing; Yang Liu
Rico is a dataset of Android app designs and a runtime mining system that needs no access to or modification of app source code. Crowd workers used free Google Play apps through a web interface to a device farm, recording interaction traces. An automated crawler then replayed each trace and kept exploring, using a screenshot-and-resource-id similarity heuristic to identify new UI states. The paper reports 10,811 traces and 72,219 unique UIs from 9,772 apps in 27 categories. Each UI has a screenshot, augmented view hierarchy, explored interactions, transition animations and a 64-dimensional autoencoder layout vector.
A dataset and collection system, not a reconstruction method: the paper proposes UI code generation and interaction modeling as possible uses but does not generate code or reconstruct apps. It records UIs and interactions reached during human sessions capped at 10 minutes, each followed by one hour of automated crawling, not complete app behavior, backend logic or source code. Multimedia, productivity and personalization categories were excluded, and only free apps were mined. Coverage was evaluated on 10 popular apps by counting unique UIs, a measure that depends on the similarity heuristic (6% false-positive and 3% false-negative rates on 1,044 labeled pairs). The demonstrated application is autoencoder-based layout similarity search with qualitative examples. The 2017 snapshot will age as apps change.
Authors & citation
Biplab Deka; Zifeng Huang; Chad Franzen; Joshua Hibschman; Daniel Afergan; Yang Li; Jeffrey Nichols; Ranjitha Kumar
DeGuard reverses layout obfuscation in Android APKs, predicting original names for the fields, methods, classes and packages that ProGuard renames. Built on Soot, it represents an app as a dependency graph over program elements and keeps Android API and other unobfuscated names fixed. It then runs MAP inference with Nice2Predict on a conditional random field learned from 1,684 F-Droid apps, under constraints that keep the renamed APK semantically equivalent. On 100 held-out F-Droid apps obfuscated with ProGuard, names identical to the originals rise from 13.3% to 79.1%, and third-party library prediction reaches 91.3% precision and 91.0% recall.
Only layout (identifier) obfuscation is addressed. Data, control-flow and cryptographic obfuscation, custom string encryption, reflection and goto-based code reordering are not reversed, and the paper notes that most malware samples it inspected also used such techniques. The benchmark is 100 benign, open-source F-Droid apps, obfuscated by enabling ProGuard without changing its rules and scored by exact match to the original names, so semantically close names count as errors; elements that ProGuard removed are not scored. Accuracy on other obfuscators or on malware may be lower. Libraries are identified by a package-name heuristic covering 133 libraries, and the training apps may import the same libraries. The 49-family malware study gives manual examples, such as Base64 decoders and location handling, without precision figures.
Authors & citation
Benjamin Bichsel; Veselin Raychev; Petar Tsankov; Martin Vechev
Reconstructs native Android UI code from screenshots or design bitmaps using OCR, computer vision, and layout heuristics; evaluates visual similarity and view hierarchies on 488 screenshots from over 100 Android and iOS applications.
The output is Android UI code; iOS screenshots are inputs, not evidence of native iOS output. Original logic and backend behavior are not recovered, and multi-screen transitions were future work. OCR/layout heuristics can misidentify content. The linked REMAUI v2 repository has older dependencies and is not the exact verified 2015 artifact.
SwiftHand generates test inputs for Android apps while learning an approximate GUI model during testing. It instruments Dalvik bytecode to read the GUI component tree and derive each screen's enabled touch, scroll, text, Back and Menu inputs; screens with the same enabled inputs are treated as equivalent states of an extended deterministic labeled transition system. The learned model steers execution toward unexplored frontier states and avoids restarts, since reinstalling an app took about 30 seconds. On 10 F-Droid apps, the authors report higher branch coverage than random and L*-based testing within the time budget, reached faster.
A test-generation technique whose learned model is an approximate abstraction for guiding exploration, not a complete or exact GUI specification or a reimplementation of the app. States are identified only by their sets of enabled inputs and merged heuristically; tab views and similar designs caused state explosion and low coverage. The implementation rejects apps with a native main entry, works correctly only on Android 4.1 or later, and cannot handle apps that store data on remote servers, so such apps were excluded from the benchmark. Evaluation covers 10 F-Droid apps under Android 4.1.2: eight ran on emulators with a 3-hour budget and two on a Galaxy Nexus phone with 1 hour. Random testing caught up on some small apps given more time. The 2013 tool is not shown to run on current Android releases.
GUI ripping reverse engineers a test model from a running desktop application rather than from its source code or specifications. Starting from the top-level windows, a ripper opens each child window by programmatically clicking executable widgets, records every window’s widgets, properties and values, and builds a GUI forest of windows. From that forest it derives event-flow graphs, which record which events can follow one another within a component, and an integration tree of component invocations. The paper describes rippers for Microsoft Windows applications, using window handles and the Windows API, and for Java Swing applications, using Java’s AWT and Swing APIs. It applies them to WordPad, Notepad, Yahoo Messenger and four applications of the TerpOffice student office suite, with the models feeding automated test generation in the authors’ GUITAR and DART systems.
The targets are 2003-era desktop GUIs on Microsoft Windows and Java Swing, not mobile apps or web interfaces; the model assumes deterministic events on widgets with discrete property values and excludes timing-dependent, continuous-media or nondeterministic GUIs. The recovered model is a window, widget and event structure for test generation, not UI code or application logic, and the paper does not establish complete behavioral coverage. Ripping can miss windows and widgets: the Windows ripper misses widgets that bypass the Windows API and cannot distinguish modal from modeless windows, while the Java ripper cannot extract the Print dialog structure. In the case studies, testers added missed windows manually, taking 5 to 10 minutes per application, and identified top-level windows by hand for Windows applications. Results come from seven applications and the authors’ own test pipeline.
Organizes 24 MCU analysis tools into configuration recovery, emulation, fuzzing and diagnosis stages, then evaluates the runnable, pipeline-compatible subset on 4,571 filtered Cortex-M firmware images. Studies where recovered configurations and emulation artifacts fail to support later analysis.
Excludes tools requiring unavailable manuals, source or hardware context, so this is not a uniform execution comparison of all 24 tools. ISA identification, admission thresholds and sampled crash diagnosis involve empirical judgments; findings do not generalize to other architectures. Reported crashes are not confirmed vulnerabilities.
Builds a noise-tolerant pseudo execution-flow graph from one power trace per test input, using newly observed behavior as LibAFL feedback. It explores embedded firmware without access to its binary, source or instrumentation.
Requires physical power measurement, known input pins and setup calibration. Evaluation covers 15 targets on two Cortex-M platforms and two commercial devices; benchmark coverage is measured by later instrumented replay, not recovered directly from power traces. Results depend on seeds and hardware, and long traces reduce throughput. The graph is not a proven control-flow graph or recovered program. Real-device harnesses are withheld.
Authors & citation
Pouya Narimani; Kseniia Rogova; Addison Crump; Martin Mohl; Meng Wang; Ulysse Planta; Pansilu Pitigalaarachchi; Ali Abbasi
Bluezz models peripherals as stateful event-condition-action systems, coordinating interrupts, MMIO and DMA rather than treating accesses independently. It uses these models to rehost BLE firmware and reach connected protocol states for analysis and fuzzing.
The BLE evaluation covers 18 targets, with additional limited MCU examples. Hardware specifications, behavioral rules and use-case drivers require preparation and human validation.
Extends KVM with MCU instruction handling, memory mapping and interrupt modeling to rehost firmware on ARMv8-A hosts. An MMIO monitor and firmware wrapper support peripheral interaction and coverage-guided testing.
The prototype needs ARMv8-A with 32-bit compatibility and EL2 virtualization. Some coprocessor operations need added emulation, and entry-point or basic-block recovery can require manual correction. Peripheral approximations can cause false positives; evaluation covers two benchmarks and 12 firmware images, not arbitrary devices.
FirmPilot reconstructs QEMU execution environments through coordinated retrieval, planning, filesystem, persistent-state and network recovery. Proposed changes are applied as constrained artifact deltas, then checked by rerunning the firmware and fixed probes.
The main denominator is 10,033 obtainable LFwC images, not all 10,913 metadata records. Reachability under a 2,400-second budget is not whole-device equivalence; HTTP success can include redirects or authentication gates. Reported results use one LLM backend and fixed probes.
FirmCure combines static dependency analysis, iterative boot-configuration repair and runtime fault diagnosis to rehost Linux firmware in QEMU. Specialized agents apply targeted repairs and use execution feedback to recover service behavior.
Evaluation covers 21 images from ten vendors and five architectures: all opened ports, but only 19 reached full service interactivity within 45 minutes. Substituting an approximate kernel cannot preserve original-kernel or dedicated-hardware vulnerabilities. LLM reasoning loops caused residual failures. These author-reported results.
Authors & citation
Chuan Hong; Zheng Zhang; Lei Zhou; Laisong Li; Chenyifan Liu; Ze Huang; Xu Zhou; Peihong Lin
Studies satellite communication modules through hardware inspection, firmware reverse engineering, protocol analysis, partial rehosting and targeted testing. A component-level threat model connects recovered data flows to security boundaries.
Detailed assessment covers three vendors in isolated laboratory setups, not operational satellite links. The reported 28 of 64 active missions are passively inferred as plausibly vulnerable, not actively compromised; encrypted or compressed telemetry is excluded. Manual analysis cannot establish completeness, and vendor-specific findings do not generalize to all modules. LLM use was editorial, not an evaluated reversing agent.
Authors & citation
Ulysse Planta; Julian Rederlechner; Martin Strohmeier; Mathias Fischer; Ali Abbasi
FIRMWELL reconstructs the execution environment of Linux firmware services by running their initialization processes, locating blocking dependencies and repairing missing system-level behavior. It combines containerized QEMU-user emulation with a QEMU-system fallback, using process traces and binary analysis to recover resources needed by the target service.
The evaluation concerns selected network services, not complete device or hardware equivalence. HTTP acceptance uses configured response-status and error-content checks; other services need their own criteria. Coverage depends on unpacking, supported architectures, initialization heuristics and peripheral models. Symbolic path explosion, solver timeouts and complex peripheral checks remain failure modes. The paper’s recovery strategy avoids patching target binaries; successful service interaction does not establish recovery of original source code or every device function.
Authors & citation
Chuan Qin; Cen Zhang; Yaowen Zheng; Puzhuo Liu; Jian Zhang; Yeting Li; Weidong Zhang; Yang Liu; Limin Sun
2025Workshop on Binary Analysis Research (BAR) 2025Conference paper
Penguin rehosts embedded Linux firmware from an extracted filesystem by deriving an editable configuration of emulation transformations, such as file changes, pseudofiles, NVRAM values and network interfaces, from static analysis. Dynamic-analysis plugins expose failures so the configuration can be refined for each target, and a minimizer identifies which transformations a working rehosting needs.
Targets embedded Linux firmware through whole-system emulation with kernel replacement, as in Firmadyne and FirmAE; the implementation covers x86, ARM and MIPS and relies on the Linux syscall ABI. In the FirmAE comparison, success means a process binds port 80 or 443; a non-empty HTTP response is a stricter check, and the authors note neither shows full functionality. The 75% gain over FirmAE is author-reported on their own corpus of 13,649 images from vendor websites. Refinement still needs an analyst, and nondeterministic runs were repeated up to ten times. It builds an analysis environment, not a faithful device reproduction or source recovery.
Authors & citation
Andrew Fasano; Zachary Estrada; Luke Craig; Ben Levy; Jordan McLeod; Jacques Becker; Caden Kline; Elysia Witham; Cole DiLorenzo; Ali Bobi; Dinko Dermendzhiev; Tim Leek; Wil Robertson
SEmu builds microcontroller peripheral models from vendor reference manuals rather than from the firmware itself. An NLP pipeline built on the Stanford tagger and parser turns register descriptions, interrupt tables and DMA channel tables taken from the manuals into structured condition-action rules, which the emulator checks, executes and chains at run time to answer MMIO accesses and raise interrupts or DMA requests. A symbolic-execution diagnostic tool built on S2E and µEmu traces invalid states back to the rules likely responsible, so that a person can correct them. The authors also use the models for fuzzing and to check driver code against specification rules.
Rule extraction is not fully automatic. The NLP engine mishandles cross-sentence references and nested conditions, and the raw rules for the clock peripherals of all three unit-test chips broke booting, so every one of P2IM's 66 unit tests failed until those rules were revised. After diagnosis-guided manual correction (26 rules added, 21 fixed and 3 removed, against 3,602 rules extracted from 23,424 sentences), the authors report that all 66 unit tests and 17 additional samples were emulated. Users must identify the MCU, configure memory maps and copy the needed manual sections into the pipeline, and diagnosis needs prepared test firmware; three developers took about two hours on average to fix the F103 I2C rules. The prototype covers 26 peripherals on five boards from three vendors (STM32F1, F4 and L1, NXP K64 and Atmel SAM3X), and trace-fidelity comparisons used real-hardware traces from only the K64F and F103RB boards. Rules model documented behaviour and do not establish equivalence with the silicon.
Authors & citation
Wei Zhou; Lan Zhang; Le Guan; Peng Liu; Yuqing Zhang
Rehosts monolithic ARM Cortex-M firmware for fuzzing, using locally scoped symbolic execution to model memory-mapped I/O accesses and focus input mutations on values relevant to firmware logic.
The Cortex-M3/M4 artifact requires memory-map configuration and does not automatically model DMA. Abstracting hardware can produce crashes that are impossible on the physical device. This is an analysis environment, not original-source recovery or faithful full-device reconstruction.
Authors & citation
Tobias Scharnowski; Nils Bars; Moritz Schloegel; Eric Gustafson; Marius Muench; Giovanni Vigna; Christopher Kruegel; Thorsten Holz; Ali Abbasi
2021Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications SecurityConference paper
ECMO transplants peripheral drivers into embedded Linux kernels to boot firmware in QEMU. It evaluated 815 firmware images across 20 kernel versions, 37 device models, and 24 vendors; 710 reached a user-space shell.
The evaluation covers embedded Linux firmware across differing SoCs, QEMU kernel rehosting, kernel crash analysis, rootkit forensics, and kernel fuzzing.
Authors & citation
Muhui Jiang; Lin Ma; Yajin Zhou; Qiang Liu; Cen Zhang; Zhi Wang; Xiapu Luo; Lei Wu; Kui Ren
Jetset rehosts firmware only as far as an analyst-chosen goal address, such as the point where a fuzzer can inject input. Given the architecture, the MMIO address range, an entry point and the goal, it symbolically executes the firmware with angr against a modified QEMU, treating peripheral reads as symbolic, cycling interrupts, and steering toward the goal with CFG distances and Tabu search. Solved constraints become C peripheral models for QEMU that replay the inferred values and interrupts so the firmware boots concretely to the goal. The authors evaluate 13 targets across x86, ARM and Motorola ColdFire, including Linux on a Raspberry Pi 2, a BeagleBoard-xM bootloader, a SEL-751 relay, a Boeing 737 CMU-900 and nine P²IM firmware, and use the CMU model with AFL to find a privilege escalation in its VRTX kernel.
This is targeted rehosting, not full device emulation or firmware decompilation. The analyst supplies the MMIO range, entry point and goal address, and the synthesized models replay one satisfying path and then the last satisfying value for each address, so they need not match real hardware and may mishandle stateful or complex peripheral interaction after the goal. DMA is unsupported; the two targets that needed it relied on an existing QEMU device model or a manually symbolic memory region. Incomplete CFGs, for example across virtual calls or x86 task switches, weaken the distance heuristic, and the BeagleBoard-xM emulator crashed when synthesized serial data was executed as code. Fidelity checks compare against hardware or a manually built QEMU model for selected targets. The VRTX privilege escalation is described by the authors as primarily of academic interest.
Authors & citation
Evan Johnson; Maxwell Bland; YiFei Zhu; Joshua Mason; Stephen Checkoway; Stefan Savage; Kirill Levchenko
µEmu emulates ARM Cortex-M microcontroller firmware without real hardware or hand-written peripheral models by symbolically executing the image with unknown peripheral registers as symbols. Invalidity-guided exploration detects invalid states such as unexpected infinite loops and steers execution toward valid paths, caching the chosen responses at individual peripheral access points in a tiered knowledge base whose rules range from simple storage to PC- and calling-context-based matching. During dynamic analysis the knowledge base answers peripheral reads while an integrated, modified AFL fuzzer supplies data-register input. The prototype extends S2E 2.0 with Cortex-M support.
Targets ARM Cortex-M firmware; the repository lists DMA and other architectures as unsupported help-wanted items. Responses only have to satisfy the firmware's own checks rather than reproduce real peripheral values, so successful emulation does not establish hardware equivalence. Invalidity checks can miss unhandled error paths, data registers can be mis-categorized (3 of 43 were missed against ground truth taken from chip manuals), and long loops whose concrete counters depend on symbols may go undetected. On P2IM's 66 unit tests the final paper reports 63 passes (95%) without manual help, and all after adding one invalid program point to each failure. Across 21 real-world firmware samples, 6 needed a user-defined invalid point and 3 a manually added data register. Coverage counts QEMU translation blocks against IDA-counted basic blocks, following P2IM.
Systematizes rehosting: moving embedded firmware from its physical hardware into a virtual environment that models enough hardware behavior for dynamic analysis such as fuzzing and symbolic execution. The authors measure why whole-system emulation does not scale using 2,574 hardware configurations, 1,956 device tree blobs from Linux 5.11.4 and 618 vendor ARM Cortex SVD files. They compare corpus CPU models with QEMU support, count unique peripherals, use driver source lines as a complexity proxy, and run a Monte Carlo simulation of adding peripheral models SoC by SoC. Supporting a random 10% of the SoCs would need about 3,947 peripherals for the Linux corpus and 1,730 for the ARM Cortex corpus, against 1,083 implemented in QEMU 5.2. The paper then classifies prior systems by how they emulate, replace, symbolically model, pass through to hardware or leave each layer unmodified, describes an iterative rehosting process with fidelity evaluation, and sets out open problems.
This is a systematization with a measurement study, not a new rehosting tool. The measurements use hardware descriptions rather than firmware images: device trees cover Linux-based (Type-1) systems and SVD files cover only ARM Cortex hardware, and only architectures or vendors with ten or more samples are counted. Driver source lines are an explicitly rough proxy for the effort of modeling a peripheral, and closed-source drivers are assigned an architecture average. The conclusion that manually written hardware emulators will not scale rests on these proxies and on QEMU 5.2 and OVPsim as of early 2021. The authors note that rehosting fidelity lacks a formal definition and that no large-scale fidelity evaluations exist, so the taxonomy compares approaches qualitatively; a concurrent survey covers fidelity in more detail.
Authors & citation
Andrew Fasano; Tiemoko Ballo; Marius Muench; Tim Leek; Alexander Bulekov; Brendan Dolan-Gavitt; Manuel Egele; Aurélien Francillon; Long Lu; Nick Gregory; Davide Balzarotti; William Robertson
Conware builds reusable software models of hardware peripherals from recordings of real firmware–peripheral interactions. An LLVM pass instruments Arduino source code to log MMIO reads, writes and interrupts on the real board, with existing hardware-in-the-loop recording as an alternative. Each recording becomes a per-peripheral graph whose read values are summarized as simple sub-models such as patterns; graphs are merged and optimized into automata that can be combined across peripherals and run in an Avatar²-based emulator. The authors model ten peripherals on an Arduino Due (SAM3X Cortex-M3) and emulate an uninstrumented smart-door-lock firmware from models trained only on vendor example code.
Training needs the real board with the peripherals attached, plus source code for the LLVM-based recorder. The evaluation covers bare-metal Arduino Due firmware with ten hobbyist peripherals, and port-mapped I/O systems are out of scope. Models encode only behaviour observed during recording, assume that state transitions follow MMIO writes, and attach interrupts to transitions rather than to the writes that trigger them, so read-triggered transitions and exact interrupt timing are not supported. The unseen-firmware result is a single door-lock program built from six peripheral models. Its UART output matched real hardware, and it executed 362 of the 609 basic blocks (59%) and 58 of the 68 functions (85%) reachable from loop(); the authors measure survivability of execution rather than full peripheral fidelity. Record-and-replay comparisons ran each sample once for 10 minutes, since replays are deterministic.
Authors & citation
Chad Spensky; Aravind Machiry; Nilo Redini; Colin Unger; Graham Foster; Evan Blasband; Hamed Okhravi; Christopher Kruegel; Giovanni Vigna
Snipuzz fuzzes IoT firmware as a client, using device responses to infer message snippets that guide mutations. It was evaluated on 20 real-world IoT devices and compared with four IoT fuzzing approaches.
The evaluation covers 20 popular real-world IoT devices and comparisons with IoTFuzzer, BooFuzz, Doona, and Nemesys. It reports five newly discovered zero-day vulnerabilities, three exposed only by Snipuzz.
DICE adds DMA input emulation to microcontroller firmware emulators that model neither peripherals nor DMA controllers. It watches firmware writes to the memory-mapped DMA controller region for pairs of source and destination transfer pointers, and infers the transfer direction from later firmware accesses. When firmware reads a peripheral-to-RAM destination buffer, DICE answers with analyzer-supplied input such as fuzzer data, growing buffer bounds conservatively. Prototypes patch the QEMU forks of P2IM (ARM Cortex-M) and a PIC32 emulator (MIPS M4K/M-Class). On 83 sample firmware covering 9 DMA controllers from 5 vendors, the authors report 33 of 37 input channels detected with no false positives, correct input for 21 of 22 buffers and 3.4% average overhead. Fuzzing 7 real firmware for 48 hours found 5 previously unknown bugs.
Emulates DMA input channels only; it does not model peripheral behavior, recover peripheral semantics or produce a faithful device replica, and supplied inputs are analyzer data rather than real peripheral traffic. It assumes transfer descriptors are written to DMA controller registers, so descriptors stored in RAM are unsupported. Destination-only configurations, used by two controller models such as nRF52832 EasyDMA, are also missed, accounting for 7 of 52 stream configurations. Buffer sizes are heuristic estimates. Ground truth came from datasheets and firmware inspection, and results cover the authors' 83 sample firmware and 7 fuzzed firmware. Bugs were found in two of the seven fuzzed firmware with a basic memory sanitizer. The PIC32 prototype runs only firmware whose other peripherals can be handled by simple default register handlers.
2021Proceedings 2021 Network and Distributed System Security SymposiumConference paper
The paper uses a POSIX-based portable MCU and replaceable peripheral drivers to rehost MCU software for native execution and security testing on commodity hardware.
It rehosts nine MCU operating systems, including Amazon FreeRTOS, ARM Mbed OS, Zephyr, and LiteOS. AFL and ASAN testing found 28 previously unknown bugs; five received CVEs and 19 were confirmed by vendors at the time of writing.
Authors & citation
Wenqiang Li; Le Guan; Jingqiang Lin; Jiameng Shi; Fengjun Li
Extends Firmadyne with empirically derived interventions for boot, networking, NVRAM, kernel behavior, and other runtime failures, enabling Linux router and IP-camera firmware web services to run for dynamic analysis.
Arbitration deliberately changes execution and targets functioning web interfaces, not identical physical-device behavior. The evaluation covers ARM little-endian and MIPS little-/big-endian firmware from routers and IP cameras; heuristics may fail on unseen devices or configurations. It reuses firmware binaries rather than recovering their source.
HALucinator rehosts monolithic ARM Cortex-M firmware by identifying hardware-abstraction-library functions and replacing them with authored Python handlers and peripheral models. LibMatch locates library interfaces; the QEMU/Avatar2 implementation supports interactive I/O, while the paper-associated HAL-fuzz variant uses AFL-Unicorn for fuzzing.
LibMatch requires an available HAL/library and a sufficiently similar compilation environment; compiler variation and inlining limit matching. Rehosting also needs a QEMU-supported architecture, firmware memory layout, interception addresses and authored handlers/models. Unmodeled direct hardware accesses may return zero or ignore writes, and simplified timing and peripheral behavior do not establish hardware equivalence. The authors’ rehosting and fuzzing demonstrations are not universal firmware coverage, complete device reconstruction or original-source recovery.
Authors & citation
Abraham A Clements; Eric Gustafson; Tobias Scharnowski; Paul Grosen; David Fritz; Christopher Kruegel; Giovanni Vigna; Saurabh Bagchi; Mathias Payer
Presents a manual framework for reverse engineering communication protocols in embedded Linux IoT systems. It applies the framework to several systems and details a WeMo smart plug case study.
The detailed evaluation covers a WeMo smart plug: extracting firmware from flash, analyzing it statically and dynamically, and examining network traffic. The framework covers read-only and writable embedded Linux filesystems.
2020Proceedings of the 13th ACM Conference on Security and Privacy in Wireless and Mobile NetworksConference paper
BaseSAFE partially rehosts closed-source cellular baseband firmware for instrumented fuzzing and uses a sanitizing allocator to detect heap buffer overflows.
The evaluation fuzzed rogue-base-station-accessible parsers in a Nucleus RTOS-based MediaTek baseband, using LTE signaling message test cases. It found memory corruption and reached about 15,000 test cases per second in total.
Frankenstein emulates Bluetooth firmware dumps and fuzzes a virtual modem. It evaluates Broadcom and Cypress stacks, finding three zero-click vulnerabilities, host escalation issues, and BlueFrag on Android.
Firmadyne crawls, extracts and emulates Linux-based firmware for network-connected commercial devices such as routers and IP cameras. It replaces the vendor kernel with pre-built, instrumented kernels for little-endian ARM and little- and big-endian MIPS, boots the extracted filesystem under QEMU full-system emulation with a userspace NVRAM library, infers the network configuration, then runs automated analyses including 60 known and 14 newly found exploits. From 23,035 images across 42 vendors, the authors extracted 9,486 and report 887 of the 1,971 network-reachable images vulnerable to at least one sampled exploit.
The work targets Linux-based firmware on little-endian ARM and little- and big-endian MIPS, which made up 90.8% of the dataset; x86, VxWorks and other real-time operating systems are outside the system. Its pre-built kernels do not load vendor out-of-tree kernel modules, so vendor kernels and modules cannot be checked, and instrumented emulation is not identical device behavior. The emulation funnel narrows sharply: of 8,591 images that initially emulated, 2,797 had a network configuration inferred and 1,971 answered ping. NVRAM semantics, interface naming and a single emulated Ethernet device on ARM are among the reported causes. WAN and LAN ports are not distinguished, so remote exploitability is not established, and results count firmware images rather than unique products and are skewed by vendor composition. FirmAE extends Firmadyne to address such emulation failures; these 2016 rates are not current emulation figures.
Authors & citation
Daming D. Chen; Manuel Egele; Maverick Woo; David Brumley
The paper emulates firmware images and uses static and dynamic tools to find vulnerabilities in embedded web interfaces. It evaluated 1,925 images from 54 vendors and found vulnerabilities in 185 images.
The study covers embedded web interfaces and network services in COTS device firmware, including routers, modems, VoIP phones, and IP/CCTV cameras. Its method is designed for different vendors, devices, and architectures.
Authors & citation
Andrei Costin; Apostolis Zarras; Aurélien Francillon
Avatar runs embedded firmware in an instrumented emulator, S²E on QEMU, while forwarding memory-mapped I/O to the real device and injecting the device's interrupts back into the emulator, so firmware that depends on undocumented peripherals can still be analyzed dynamically. The device is reached through JTAG via OpenOCD or through a small debugging stub over a serial line. Because forwarding every access over a 38400-baud serial link is very slow, about five instructions per second in one of the authors' measurements, Avatar also keeps memory that never behaves like I/O local to the emulator, migrates selected code to run natively on the device, and switches state between the two. With S²E's selective symbolic execution, the authors probe a hard-disk mask-ROM bootloader for hidden commands, an IEEE 802.15.4 sensor node for an injected overflow, and the SMS decoding routines of a GSM feature phone.
Avatar needs the physical device and a working debug channel or an injectable stub; it does not model peripherals, so it is not standalone rehosting. The three case studies are all ARM devices (a hard disk with an ARM966 core, a Redwire Econotag and a Motorola C118 phone), and each needed manual work to choose hook points and code regions. Full-separation mode is too slow for time-critical code such as DRAM initialization, and the authors report that memory changed by DMA, or by code migrated to the device, is hard to keep synchronized with the emulator. The sensor-node vulnerability was artificially inserted into a demonstration program, the bootloader study confirmed documented commands rather than finding a backdoor, and the phone's GSM stack was too complex for symbolic execution: 42 subroutines were explored without an exploitable condition before state explosion. These are 2014 results on an S²E port the authors extended for ARM.
Authors & citation
Jonas Zaddach; Luca Bruno; Aurélien Francillon; Davide Balzarotti
2026Proceedings of the ACM on Software Engineering / FSE 2026Journal article
Transfers field-boundary patterns from standardized protocol traces labeled by Tshark to poorly documented target protocols. A source-selection stage chooses related protocols, and a neural domain-adaptation model infers message syntax from their shared patterns.
Recovers message syntax, not state transitions. Evaluation uses 12 protocol variants; payload noise can distort source selection, while encrypted, compressed or obfuscated traffic needs separate preprocessing. Camera experiments supplement format metrics, but the four reported issues were vendor-known and do not establish complete protocol recovery.
Tests reasoning over protocol state machines using 1,482 questions across 16 protocols. Four context settings distinguish RFC retrieval from reasoning over supplied transitions; tasks cover next states, conditions, predecessor states and paths.
Evaluates reasoning, not a new FSM extractor or implementation reconstruction. Gold FSMs were prepared by one author; judge agreement was checked on 50 instances by the three authors. Packet formats, error handling, configurations and real code mappings are outside scope. Results depend on context, judges and output budgets.
PVParser recovers process-variable layouts from industrial traffic and historian records. Periodic-pattern detection locates relevant payloads; a modified Monte Carlo tree search proposes field boundaries, types and endianness, then aligns decoded sequences with observed process variables.
Payload localization uses SCADA-N, SWaT and WADI; field/semantic recovery uses SWaT and WADI. Recovery depends on traffic and historian observations plus recurrent, stable plaintext byte layouts. Encrypted traffic, bit-level buses and event-driven messages whose PV lists vary are outside scope. It complements protocol dissectors.
2025Proceedings of the ACM on Software Engineering (ISSTA 2025)Journal article
Infers message-field boundaries by following concrete message-guided paths through binary protocol parsers, then symbolically replaying them to extract byte-level data flow. It merges bytes using value-preserving operations and similar processing traces. For the 1,000-message datasets of four public ICS protocols, the authors report average boundary F1 of 0.76 and field perfection of 0.67.
Recovers message syntax, not field semantics, protocol state machines or endpoint equivalence. It needs the binary, loader configuration and correctly identified initialization, input and parsing functions; manual filtering remains necessary, especially without symbols. Encrypted protocols and bit-level fields are unsupported, and ignored or only copied bytes can be merged incorrectly. Perfection measures fields with both boundaries correct. Quantitative tests use four public protocols with 50- and 1,000-message sets; separate proprietary studies cover three protocols in five firmware targets, with interaction checks on three devices and manually established ground truth. Those checks do not demonstrate complete device behavior.
ProtocolGPT infers protocol state machines from implementation source code with a retrieval-augmented LLM. It extracts state-machine-related code with regular expressions and a code filter, splits it into syntax-aware chunks stored in a FAISS vector store with OpenAI embeddings, and prompts GPT-4 step by step to recover states and message-triggered transitions. On six protocols in five open-source code bases (IKEv2, TLS 1.3, TLS 1.2, BGP, RTSP and L2TP), the authors report transition precision above 90%, more than 30% higher precision and 55% higher recall than baselines. Seeding AFLNet with the inferred machines raised coverage by more than 20% and found two previously unknown crash bugs in an RTSP server.
Requires source code; it does not work from binaries or traffic alone. Each implementation was chosen for explicitly defined states, and results cover only the six evaluated protocols, two of which share the s2n-tls code base. Ground truth was built manually by three experts from repository code; precision and recall count only fully correct transitions. RFCNLP had higher precision on RTSP. Results depend on GPT-4 at temperature 0.2 through a commercial API, so outputs and costs may change. The inferred machines are LLM summaries, not proofs of implementation behavior or equivalence. Fuzzing gains come from AFLNet seeds; BGP's event-driven transitions limited message-based coverage. The two crash bugs are in the feng RTSP server.
Authors & citation
Haiyang Wei; Ligeng Chen; Zhengjie Du; Yuhan Wu; Haohui Huang; Yue Liu; Guang Cheng; Fengyuan Xu; Linzhang Wang; Bing Mao
Infers protocol-message field boundaries and semantics from instrumented server binaries and input messages. BinPRE groups bytes by the instruction operators that process them, applies detectors for field types and functions, then refines results across similar messages. On eight protocols with 50 messages each, the authors report average field perfection of 0.73 and type/function F1 scores of 0.74/0.81.
This is field inference, not recovery of a protocol state machine or complete endpoint behavior. It requires an accessible, instrumentable implementation and runtime message processing; passive captures alone are insufficient. Perfection counts exactly segmented fields against all true fields, not fully recovered protocols. The detector library covers five semantic types and six functions. Byte-level taint cannot accurately resolve bit flags, and unexercised behavior is not established. Ground truth uses Wireshark dissectors and specifications; three binary-analysis baselines were author reimplementations. Reported fuzzing improvements are a downstream evaluation, not proof of implementation equivalence.
DynPRE starts from captured client–server traces and then interacts with a live protocol server to infer message formats and types. It detects session-specific identifiers such as cookies and rewrites them on the fly so that replayed requests stay valid. It then sends byte-level probe requests whose responses serve as semantic feedback for field identification, aligns the responses, and applies a request–response mapping refinement that also recognizes message types. On 12 protocols with 10-, 100- and 1,000-message datasets, the authors report average field perfection of 0.50 and message-type V-measure of 0.94, against 0.15–0.31 perfection for Netzob, NetPlier, FieldHunter, BinaryInferno and NEMESYS.
The method needs captured traces plus a reachable server that it can query repeatedly; it does not work from passive captures alone. Probing changes server state: each probe opens a new session, but protocols with global state, such as IoT device power state, can still degrade feedback. On the 1,000-message datasets an analysis averaged 259 minutes, 50,344 connection attempts and about 3.98 million exchanged messages, so slow or rate-limiting servers reduce efficiency. Encryption and authentication were disabled in the open-source evaluation setups, and encrypted traffic in the IoT case studies was decrypted with device private keys. Checksummed protocols need an added checksum-reversal step, shown only on ICMP. Format ground truth comes from tshark dissectors, and message types from manual reading of protocol documents. The five-device IoT case studies test whether generated messages trigger device behaviors, not whether specifications are complete. State-machine inference reuses existing methods and is not evaluated as a contribution.
NetLifter statically lifts source-available C protocol parsers into BNF-style message grammars and first-order constraints between packet fields. It analyzes annotated LLVM bitcode using abstract format graphs, localized path-sensitive analysis, unfolding and reordering. The resulting field boundaries, constraints and names can support packet generation and parser construction.
Requires source/LLVM IR and packet-buffer/length annotations; it is not closed-binary or traffic-only recovery. The paper targets common top-down C parsers and does not support C++, inline assembly or libraries without available code. Pointer/loop analysis remains approximate; checksum fields can be identified while some checksum constraints are omitted, and fields ignored by an implementation may be missed. Precision/recall was manually checked on ten selected protocols. The reported fuzzing evaluation used three-hour runs repeated twenty times and bypassed checksum checks. It infers formats, not protocol state machines or proof of implementation equivalence.
The paper extracts protocol message formats from parser code by analyzing parsing loops as state machines. It evaluates format inference and the use of the extracted machines in protocol fuzzers.
The analysis targets protocols with constraint-enhanced regular-expression formats parsed by finite-state machines. Reported results include over 90% precision and recall in five minutes, 20%–230% higher fuzzer coverage, and ten additional zero-days detected.
Infers partial binary-message formats from static message samples without instrumenting an executable. Specialized detectors recognize floats, timestamps, lengths, entropy boundaries and serialization patterns; a graph-based integration step reconciles their descriptions. Across samples from ten protocols, the authors report top-level field-boundary precision of 0.69, recall of 0.73 and false-positive rate of 0.04 for the 1,000-message setting without analyst priors.
Assumes error-free, uncompressed, unencrypted, byte-aligned data and infers common fields or payload regions beginning at consistent offsets. It leaves optional/union fields and unsupported serialization patterns unresolved; sample diversity matters. The 26 encapsulated-payload test sets were pre-grouped using ground-truth message types, not discovered automatically. Ground truth merges adjacent fields that are constant across the sample; empty inference results are excluded from reported averages where stated. BI+ additionally uses supplied endianness and capture-time ranges. These are field-boundary results, not complete semantic recovery, protocol-state inference or behavioral equivalence.
StateInspector learns protocol Mealy machines by combining active message queries with snapshots of runtime memory, without requiring the target’s source code. It identifies candidate state-defining heap locations and uses taint and concolic analysis to merge states, evaluated by the authors on TLS 1.2 and WPA/2 implementations.
Assumes a finite Mealy-machine abstraction, a known input alphabet that can reach the relevant states, and state-defining memory on the heap exercised by bootstrap queries. State merging depends on configured search-depth and symbolic look-ahead bounds. Missing inputs or state stored elsewhere can leave behavior unmodeled; malformed harness messages, response abstraction and timeouts can create misleading transitions. Learned models and bounded checks do not establish full implementation equivalence, recover unknown message formats or reconstruct an application’s source code.
Authors & citation
Chris McMahon Stone; Sam L. Thomas; Mathy Vanhoef; James Henderson; Nicolas Bailluet; Tom Chothia
2022Formal Methods in System DesignJournal article
Learns Mealy-machine models of Bluetooth Low Energy implementations on physical peripherals by active automata learning. An nRF52840 central running SweynTooth-derived firmware sends abstract BLE inputs, with error handling, repeated queries and caching for lost packets and non-deterministic replies. The authors learn connection-procedure models of six devices and legacy-pairing models of three, and use model differences to fingerprint their SoCs.
Models are bounded by the chosen abstract alphabet and cover only connection and legacy pairing; message formats are not inferred, and secure pairing is left to future work. Equivalence was approximated with a small number of conformance tests per state, so models are hypotheses, not proofs. One SoC was learnable only with reduced alphabets, two were learned from an already-connected state, and some pairing runs needed manual hard resets. The fingerprinting sequence holds only for the tested SoCs. Apart from one crashing input sequence, the authors report no security vulnerabilities in the learned models.
2022IEEE Symposium on Security and Privacy 2022Conference accepted
Extracts protocol FSMs from RFC prose using technical word representations, zero-shot text mapping, and rules, then uses the extracted FSMs to synthesize attacks.
DIKEUE checks 4G LTE control-plane (NAS and RRC) implementations in commercial phones and modems for noncompliance through differential testing rather than predefined security properties. Its FSM inference module runs TTT active automata learning in LearnLib through an LTE-specific adapter on a modified srsLTE network with a USRP B210 radio, adding a context checker, query cache, inconsistency resolver and reset without rebooting. An equivalence checker then encodes pairs of learned input-output finite state machines for the nuXmv model checker and enumerates diverse deviation-inducing input sequences, which the authors analyze manually.
Covers control-plane NAS and RRC procedures with a chosen input alphabet and queries up to length 12; session management, data-plane protocols and 5G are left to future work. Learned machines are hypotheses bounded by Wp-method conformance testing and custom termination conditions. The authors list false-negative sources, including omitted message predicates, pruned infeasible sequences, models disconnected by removing deviating transitions, and devices that are noncompliant in the same way. Findings cover 14 devices with basebands from five vendors (Qualcomm, Intel, MediaTek, HiSilicon and Exynos): 17 deviations, 15 of them new, of which the authors class 11 as exploitable and 3 as potential interoperability issues. Deviations were triaged and attacks constructed manually, and the results do not establish the behaviour of other devices, firmware versions or networks.
Authors & citation
Syed Rafiul Hussain; Imtiaz Karim; Abdullah Al Ishtiaq; Omar Chowdhury; Elisa Bertino
Uses multiple sequence alignment and probabilistic keyword-field inference to cluster network messages by type, evaluated across ten protocols with downstream format and state-machine analysis.
Requires unencrypted traces and cannot recover unseen message types. Larger alignments, sub-byte keywords and one-way traffic can reduce accuracy. Format/FSM inference is downstream evaluation rather than the main contribution; state semantics in a case study were manually annotated. These models do not establish complete application/device reconstruction.
LibreCAN reverse-engineers proprietary CAN message formats from traffic recorded at a vehicle's OBD-II port, without the OEM's DBC translation files. Phase 0 extracts candidate signal boundaries from bit-flip behaviour, extending the earlier READ algorithm; Phase 1 correlates extracted signals with standard OBD-II PIDs and smartphone IMU data to identify kinematic signals and estimate their scale and offset; Phase 2 filters the CAN IDs recorded while a driver performs 53 scripted body-related events to shortlist candidate IDs for each event.
Evaluated on four newer vehicles from a single OEM, using DBC files the authors obtained as ground truth and their own judgement of which CAN signals correspond to each reference signal; they call the evaluation partly subjective because the OEM was not involved. Phase 0 recovered 51.6–62.4% of DBC signals, but only 21.0–36.4% of extracted signals had correct boundaries, partly because normal driving rarely exercises high value ranges. With extracted boundaries, Phase 1 precision was 61.1–78.1% and recall 25.6–57.3%; Phase 2 cut candidates more than tenfold, with 8.5–15.0% precision. Only buses reachable through OBD-II were analysed, so signals on other buses were missed. The system reads traffic only and recovers a partial translation table, not complete DBC files, ECU logic or validated injection.
Authors & citation
Mert D. Pesé; Troy Stacer; C. Andrés Campos; Eric Newberry; Dongyao Chen; Kang G. Shin
201913th USENIX Workshop on Offensive Technologies (WOOT 19)Conference paper
Infers message formats of proprietary wireless protocols from demodulated physical-layer captures: it aligns messages by preamble and sync word, then uses RSSI and dedicated engines to label length, address, sequence-number and checksum fields, optionally using known addresses or field positions.
Only message formats are inferred: preamble, sync, length, address, sequence number and checksum. State-machine inference and payload fields, such as an on/off bit, are out of scope. The method assumes length-efficient binary messages with one preamble and sync word each. Accuracy is measured on 8 generated protocols; 3 real smart-home captures (EnOcean, Homematic BidCoS and an eQ-3 protocol used by RWE Smarthome) serve for performance tests, which the authors call a limited test set.
201812th USENIX Workshop on Offensive Technologies (WOOT 18)Conference paper
Infers approximate field boundaries within individual binary-protocol messages from changes in bit congruence, and introduces Format Match Score to compare inferred segmentation with known protocol dissectors.
Requires plaintext payloads. Evaluation covers DNS, DHCP and NTP with protocol-tuned parameters. Single-byte fields are not inferred; zero-filled fields, character sequences and pseudo-random values can cause errors. FMS needs known field boundaries for validation. The output is approximate segmentation, not field semantics, a state machine or complete application/device behavior.
Infers the vocabulary of an undocumented protocol, its message types and formats, from communication traces. The approach uses contextual information embedded in the messages to improve message clustering and the identification of field boundaries; the HAL record lists contextual clustering and semantic sequence alignment as keywords. The authors motivate the work with parsers and traffic generators for network security products, and compare it with their own reimplementations of ASAP, Discoverer and ScriptGen. The first two authors are the authors of Netzob, the open-source protocol reverse-engineering framework.
The method starts from captured traces and addresses the protocol’s vocabulary, not its grammar or state machine; contextual hints help only where such information appears in the observed messages. Baseline results come from the authors’ reimplementations of ASAP, Discoverer and ScriptGen rather than the original tools. The comparative study and its results are the authors’ 2014 evaluation. Netzob has changed substantially since 2014.
Prospex infers message formats and a protocol state machine from execution traces of a Linux server handling client sessions. Dynamic taint analysis splits each session into messages and recovers their formats; messages are then clustered into types by input-format, execution and server-impact similarity, such as the system calls, outputs and file paths a message triggers. The observed type sequences form an augmented prefix tree acceptor, which prerequisite and end-state heuristics label before Exbar computes a minimal consistent automaton. The resulting specifications are translated for the Peach fuzzer to reach later protocol states. The evaluation covers the Agobot command-and-control protocol, SMTP on Sendmail, SMB on Samba and SIP on Asterisk.
Prospex traces one endpoint, the server, and its taint tool runs only on Linux, so only Linux server binaries were analyzed. It cannot handle encrypted traffic: TLS sessions made up the 6.5% of 31,903 Postfix sessions that the inferred SMTP specification could not parse. Training sets are small (for example 16 SMTP and 31 SMB sessions), and the inferred machines cover only behavior exercised during training; two SMB states are artifacts of limited training variety, and SMB recall against the reference automaton is 0.58. Reference state machines were built manually by the authors, and soundness was checked with 50,000 random walks per protocol; the sk-strings and beams baselines are state-machine learners, not full protocol reverse-engineering systems. The fuzzing case studies found previously known issues in an older Samba release and in Asterisk, not new vulnerabilities in the latest Samba or Windows XP SMB versions tested.
Authors & citation
Paolo Milani Comparetti; Gilbert Wondracek; Christopher Kruegel; Engin Kirda
Tupni infers an input format from an execution trace of an x86 program parsing one input, using byte-level dynamic taint tracking. It segments the input into fields from the instructions that operate on tainted bytes, finds record sequences from parsing loops, groups records into types by the code that handles them, and emits an enhanced BNF grammar with length, value and some inter-message constraints. Grammars from several inputs can be merged into a more complete specification. The evaluation covers five file formats (WMF, BMP, JPG, PNG, TIF) and five network protocols (DNS, RPC, TFTP, HTTP, FTP), and the output is used for zero-day vulnerability signature generation with ShieldGen.
Tupni needs the parsing program and a recorded execution trace for every input; it does not work from captured traffic or files alone. The prototype combined iDNA traces with a re-implemented Vigilante taint engine, and the traces came from Windows XP SP2 binaries such as gdi32.dll, gdiplus.dll, nslookup.exe and IIS components. Single-input accuracy uses one input per format, scored by the authors against published specifications; WMF, DNS and HTTP guided the design, compressed image data was skipped in favor of container structure, and no quantitative baseline was run because earlier tools were unavailable. All record sequences were found, with some false-positive sequences, and 889 of 1,551 PNG fields were misidentified because the program ignored repeated chunk data. Coverage depends on input diversity, constraints can be specific to the samples, byte-level taint handles fields that do not align with bytes poorly (including many encrypted or compressed formats), and formats beyond regular expressions are out of scope. Inter-message constraints were evaluated only for TFTP, and multi-input aggregation only on 150 WMF files.
Authors & citation
Weidong Cui; Marcus Peinado; Karl Chen; Helen J. Wang; Luis Irun-Briz
Polyglot infers the format of a message that a program receives by monitoring how the program binary processes it, an approach the authors call shadowing. An execution monitor taints network input and records an instruction trace; offline analyses then find direction fields, such as length and pointer fields, from tainted memory accesses and loop conditions, separators from comparisons between tainted bytes and constants, multi-byte fixed-length fields from how instructions group input bytes, and protocol keywords from successful comparisons. It works on stripped IA-32 binaries without source code or debugging information.
It extracts the format of one received message at a time and only of messages given to the analysis; messages that never appear stay unknown, and messages the program sends are not analyzed. It does not infer the protocol state machine. The evaluation covers eleven 2007-era programs implementing DNS, HTTP, IRC, ICQ and Samba, mostly Windows servers plus an ICQ client and a Linux Samba server, with one representative message per protocol, compared against Wireshark 0.99.5 dissectors. Field formats omit data types and encodings, analysis is at byte granularity so sub-byte flags are missed, and field semantics are limited to direction fields; timestamps, checksums and addresses are not identified. Message boundaries rely on assuming a synchronous protocol, unused fixed-length fields after an ICQ length field were merged into the variable-length field, and the authors state that the techniques are not fully resistant against obfuscation.
Authors & citation
Juan Caballero; Heng Yin; Zhenkai Liang; Dawn Song