2026 · Preprint
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
UI2App evaluates runnable React/TypeScript applications generated from 600 screenshots across 95 web applications. It separates build executability, visual fidelity and screenshot-implied interaction behavior, including cross-route state, using a reference interaction taxonomy and public evaluation pipeline.
Scope and limitations
Static screenshots underdetermine behavior; the rubric accepts multiple valid implementations rather than proving equivalence to the original application or backend. Auto IIS judges generated source and screenshots without tool execution; human runtime validation covers a 45-application subset. Cross-route state is not necessarily durable persistence across reloads or sessions. Main results use one generation per application-model pair, with repeated runs on a subset.
Why it fits this collection
Tests reconstruction from observed web interfaces while distinguishing visual copying from inferred interaction logic; its explicit behavior ambiguity makes it a bounded complement to trace-guided reconstruction benchmarks.
Publication and review details
- Venue
- arXiv
- Status
- Preprint
- Publication / posting date
- July 7, 2026
- Date source
- First arXiv submission date. The reviewed v2 was submitted on 2026-10-04; it is not a separately verified conference publication.
- First arXiv submission
- July 7, 2026
- arXiv
- 2607.06306
- Reviewed arXiv version
- v2
- DOI
- 10.48550/arXiv.2607.06306
Use v2’s 95 applications and 600 screenshots, not v1’s 45/327 counts. The arXiv DOI is registered. Public code is MIT; the dataset and reference annotations are CC BY 4.0, with upstream code and third-party asset rights separate. The documented pipeline needs Python 3.10+, Node/npm, pnpm, Chromium, Tesseract and model access; Auto IIS additionally needs an authenticated Codex CLI.
Sources reviewed
Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.
Discussed in the survey
Primary sources
- https://arxiv.org/abs/2607.06306
- https://arxiv.org/html/2607.06306v2
- https://arxiv.org/pdf/2607.06306v2
- https://api.datacite.org/dois/10.48550/arXiv.2607.06306
- https://chenmancm169.github.io/UI2App-ProjectPage/
- README.md GitHub · chenmancm169/UI2App · Revision 520a59a
- LICENSE GitHub · chenmancm169/UI2App · Revision 520a59a
- requirements.txt GitHub · chenmancm169/UI2App · Revision 520a59a
- iis/evaluate.py GitHub · chenmancm169/UI2App · Revision 520a59a
- https://huggingface.co/datasets/UI2App/UI2App/blob/32e6f6193c5c265cf6d8afcbce12b641e3c2a758/README.md
- https://huggingface.co/datasets/UI2App/UI2App/blob/32e6f6193c5c265cf6d8afcbce12b641e3c2a758/annotations/rii.jsonl