2026 · Conference paper
Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering
BinJudgeBench collects expert judgments for function naming, binary-code summaries and decompilation refinement. BinJudge learns to select a judge configuration for each input and task, evaluating agreement with human scores rather than executable correctness.
Scope and limitations
The router uses five-fold validation on 1,233 annotated outputs. Initial annotators are three authors, with an independent industrial expert checking a subset. Human scores are subjective, and fluent unsupported answers can mislead judges. Correlation on these tasks neither proves program equivalence nor establishes that judges can replace human review.
Why it fits this collection
Provides evidence-oriented evaluation resources for interpreting names, summaries and refined pseudocode.
Publication and review details
- Venue
- ASE 2026
- Status
- Conference paper
- Publication / posting date
- October 7, 2026
- Date source
- Publisher-deposited print publication date; online and arXiv dates are recorded separately.
- First arXiv submission
- August 7, 2026
- arXiv
- 2608.07038
- Reviewed arXiv version
- v1
- DOI
- 10.1145/3832783.3834339
The arXiv abstract still says accepted, while ACM’s Crossref deposit records ASE 2026 publication on 2026-10-07. Direct ACM access returned HTTP 403. The paper-linked Zenodo concept 19247812 resolves to version 19310393, titled Artifact for BinJudge, with anonymous creator metadata and CC BY 4.0.
Sources reviewed
Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.