# Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

> BinJudgeBench collects expert judgments for function naming, binary-code summaries and decompilation refinement. BinJudge learns to select a judge configuration for each input and task, evaluating agreement with human scores rather than executable correctness.

- Authors: Xiuwei Shang; Li Hu; Xiao Jiang; Jieke Shi; Junda He; Zhou Yang; Shaoyin Cheng; Guoqiang Chen; Weiming Zhang; David Lo
- Venue: ASE 2026
- Status: Conference paper
- Publication / posting date: October 7, 2026
- Date source: Publisher-deposited print publication date; online and arXiv dates are recorded separately.
- First arXiv submission: August 7, 2026
- arXiv: 2608.07038
- Reviewed arXiv version: v1
- DOI: 10.1145/3832783.3834339
- Sources reviewed: 2026-10-20
- Added to the catalog: 2026-10-20
- Original paper: <https://dl.acm.org/doi/10.1145/3832783.3834339>
- PDF: <https://arxiv.org/pdf/2608.07038v1>
- Project: <https://zenodo.org/records/19310393>
- BibTeX: <https://awesomerea.org/papers/citations/paper-binjudge.bib>
- Catalog page: <https://awesomerea.org/papers/paper-binjudge/>
- Record JSON: <https://awesomerea.org/papers/paper-binjudge/record.json>
- Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-binjudge/index.md)

## Scope and limitations

The router uses five-fold validation on 1,233 annotated outputs. Initial annotators are three authors, with an independent industrial expert checking a subset. Human scores are subjective, and fluent unsupported answers can mislead judges. Correlation on these tasks neither proves program equivalence nor establishes that judges can replace human review.

## Why it fits this collection

Provides evidence-oriented evaluation resources for interpreting names, summaries and refined pseudocode.

## Publication and review notes

The arXiv abstract still says accepted, while ACM’s Crossref deposit records ASE 2026 publication on 2026-10-07. Direct ACM access returned HTTP 403. The paper-linked Zenodo concept 19247812 resolves to version 19310393, titled Artifact for BinJudge, with anonymous creator metadata and CC BY 4.0.

## Primary sources

1. <https://arxiv.org/abs/2608.07038v1>
2. <https://arxiv.org/html/2608.07038v1>
3. <https://api.crossref.org/works/10.1145/3832783.3834339>
4. <https://zenodo.org/api/records/19247812>
5. <https://zenodo.org/records/19310393>

## About this entry

Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.