# ProgramBench: Can Language Models Rebuild Programs From Scratch?

> Evaluates black-box program reconstruction across 200 tasks: agents query an executable and read its documentation, then independently implement a codebase whose behavior is checked by generated tests.

- Authors: John Yang; Kilian Lieret; Jeffrey Ma; Parth Thakkar; Dmitrii Pedchenko; Sten Sootla; Emily McMilin; Pengcheng Yin; Rui Hou; Gabriel Synnaeve; Diyi Yang; Ofir Press
- Venue: NeurIPS 2026
- Status: Conference accepted
- Publication / posting date: 2026 (exact date not verified)
- Date source: The official NeurIPS 2026 poster listing verifies conference acceptance; no proceedings publication date is asserted.
- First arXiv submission: May 5, 2026
- arXiv: 2605.03546
- Reviewed arXiv version: v1
- DOI: 10.48550/arXiv.2605.03546
- Sources reviewed: 2026-10-16
- Added to the catalog: 2026-10-17
- Original paper: <https://arxiv.org/abs/2605.03546>
- PDF: <https://arxiv.org/pdf/2605.03546v1>
- Code: <https://github.com/facebookresearch/ProgramBench>
- Project: <https://programbench.com/>
- BibTeX: <https://awesomerea.org/papers/citations/paper-programbench.bib>
- Catalog page: <https://awesomerea.org/papers/paper-programbench/>
- Record JSON: <https://awesomerea.org/papers/paper-programbench/record.json>
- Chinese version: [简体中文](https://awesomerea.org/zh/papers/paper-programbench/index.md)

## Scope and limitations

Binary inspection, decompilation, and internet access are forbidden during inference. Passing finite tests does not prove complete behavioral equivalence or recovery of the original source. Later live-leaderboard runs are separate from the reviewed arXiv results.

## Why it fits this collection

Provides an executable benchmark for recovering software behavior through experiments, complementing benchmarks that permit static binary analysis.

## Publication and review notes

NeurIPS lists a December 2026 poster; this records acceptance, not proceedings publication. The July 2026 DL4C oral was a separate non-archival workshop appearance. The DOI identifies the arXiv preprint. The pinned harness is version 1.2.5 under MIT; its generated-test dataset preserves per-task attribution and upstream license terms, including copyleft obligations for derivative portions.

## Primary sources

1. <https://arxiv.org/abs/2605.03546>
2. <https://arxiv.org/html/2605.03546v1>
3. <https://neurips.cc/virtual/2026/poster/139090>
4. <https://neurips.cc/Downloads/2026>
5. <https://dl4c.github.io/papers/>
6. <https://dl4c.github.io/callforpapers/>
7. <https://programbench.com/>
8. <https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/README.md>
9. <https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/docs/README.md>
10. <https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/pyproject.toml>
11. <https://github.com/facebookresearch/ProgramBench/blob/895f65916bc5ba34a915d36da321fde0301125ad/LICENSE>
12. <https://huggingface.co/datasets/programbench/ProgramBench-Tests/blob/de0ddfb637590c7ecb54fa0b5301f6dc7dfbcee5/README.md>

## About this entry

Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Original editorial content: MIT. Linked projects, papers and assets retain their own rights and licenses.