# CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking > CodeArt pre-trains a BERT-sized encoder from scratch on 26 million stripped x86-64 functions built from jTrans's BinaryCorp. Instead of relying on symbols, it uses IDA Pro control-flow graphs and a binary data-flow analysis to compute control and data dependences, then restricts self-attention with masks so a token attends only to instructions it transitively depends on or that depend on it, to tokens in its own instruction, and to a global [CLS] token; pre-training masks tokens and perturbs the dependence graph. The authors fine-tune it for binary similarity, malware family classification and type inference. They report improvements over prior models from 53% to 64% recall@1 for similarity in pools of 500, from 49% to 60% LRAP for malware families, and from 74% to 94% average F1 for type inference. - Authors: Zian Su; Xiangzhe Xu; Ziyang Huang; Zhuo Zhang; Yapeng Ye; Jianjun Huang; Xiangyu Zhang - Venue: Proceedings of the ACM on Software Engineering (FSE 2024) - Status: Journal article - Publication / posting date: July 12, 2024 - Date source: ACM's Crossref deposit records print and online publication on 2024-07-12. The July 2024 issue date in the reference format and the FSE 2024 conference dates are not substituted for it. - First arXiv submission: February 19, 2024 - arXiv: 2402.11842 - Reviewed arXiv version: v1 - DOI: 10.1145/3643752 - Sources reviewed: 2026-10-19 - Added to the catalog: 2026-10-27 - Original paper: - PDF: - Code: - BibTeX: - Catalog page: - Record JSON: - Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-codeart/index.md) ## Scope and limitations The model handles x86-64 only, with functions disassembled by IDA Pro and inputs capped at 512 tokens. Similarity is fine-tuned on BinaryCorp-3M and tested on six projects (Curl, Binutils, ImageMagick, SQLite, OpenSSL and PuTTY), with Coreutils for validation, against GNN, GMN, jTrans and DiEmph baselines; on PuTTY, whose functions are short, CodeArt is only comparable to DiEmph. Type inference uses the x86-64 part of StateFormer's dataset with 35 labels and compares with StateFormer's reported results rather than a rerun. Malware classification uses 5,484 VirusTotal and VirusShare samples from the 100 most frequent families, against jTrans and a re-implemented CNN. All gains are relative to these baselines and settings. CodeArt produces embeddings and task-specific predictions; it is neither a decompiler nor a malware detector. ## Why it fits this collection Adds general-purpose pre-training for stripped binary functions that uses program dependences in place of missing symbols, evaluated on similarity, type inference and malware-family tasks that support reverse engineering. ## Publication and review notes Crossref records Proceedings of the ACM on Software Engineering volume 1, issue FSE, pages 562–585, and the seven-author order matches arXiv. The ACM landing page refused automated access, so publication facts follow Crossref and the reviewed text is arXiv v1 (2024-02-19), the only version, which carries the PACMSE reference format and DOI. The abstract lists the three gains in the order similarity, type inference, malware; the contribution list and evaluation assign 49%→60% LRAP to malware family classification and 74%→94% F1 to type inference. The paper links ziansu/codeart. At the reviewed 2025-03-10 commit the repository declares MIT and its README points to an artifacts branch; README-docker.md describes a 3.8 GB Docker image with a Gradio demo and links the PurCL CodeArt-Artifacts collection on Hugging Face, whose checkpoints and datasets, including top-100 malware-family data, declare no license. The REQUIREMENTS file asks for Docker, 64 GB RAM, a 24 GB GPU and up to 100 GB of disk. ## Primary sources 1. 2. 3. 4. 5. 6. 7. 8. 9. ## About this entry Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX. Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.