2026 · Preprint
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
Studies development of the Extractor reverse-engineering tool through commits, available session logs and code-provenance tracing. Taxonomies separate instruction intent, change triggers and agent claims, exposing errors corrected before commits that ordinary repository history misses.
Scope and limitations
One codebase with human-directed work, no human-coded comparison. The 14.3% figure is 15 of 105 audited generation events, including test defects and stale tests, not a defect rate per source line or in shipped code. LLM scoring has sampled human calibration; transcripts are incomplete and the archive omits per-response evidence bundles. This companion study does not independently corroborate the related schema-recovery results.
Why it fits this collection
Adds provenance and claim-auditing methods for agent-built reverse-engineering tools.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- September 24, 2026
- Date source
- First arXiv submission; the reviewed version is v1, not a verified proceedings publication.
- First arXiv submission
- September 24, 2026
- arXiv
- 2609.29744
- Reviewed arXiv version
- v1
The author's pinned reproducibility-archive README documents lost sessions and unreproducible response-scoring evidence. The repository exposes an archive and Git-history material. No repository-wide license was verified; the project link points to the artifact landing page.
Sources reviewed
Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Primary sources
- https://arxiv.org/abs/2609.29744
- https://arxiv.org/html/2609.29744v1
- README.md GitHub · doug-leith/extractor-code-development · Revision 4f74f79
- README.md GitHub · doug-leith/ios-protobuf-decoding · Revision 9a4a688