Awesome REA
中文Star on GitHubStar1.1k stars
Star on GitHubStar1.1k stars

2025 · Conference paper

WAFFLE: Fine-tuning Multi-Modal Model for Automated Front-End Development

Shanchao Liang; Nan Jiang; Shangshu Qian; Lin Tan

WAFFLE fine-tunes multimodal LLMs to generate HTML/CSS from UI screenshots. A quarter of the decoder's attention heads use a structure-aware mask that limits each element's tokens to its parent, preceding siblings and its own earlier tokens. A contrastive objective aligns image and code embeddings over groups of four mutated WebSight-v0.1 pages. With VLM-WebSight and Moondream2 backbones, the authors report up to 9.00 pp higher HTML-Match and 0.0982 higher CW-SSIM than standard fine-tuning on WebSight-Test, plus gains on every Design2Code metric.

Scope and limitations

Generates static HTML/CSS from a single screenshot; it does not recover a site's original source, scripts, interactions, navigation or backend. HTML-Match strips styles and attributes before requiring pixel-identical renders, while CW-SSIM, CLIP and LLEM are similarity scores; the authors note that these metrics do not fully capture human judgment. Training data and the 500-page WebSight-Test are synthetic WebSight-style pages; Design2Code supplies 484 real-world pages. Only two backbones were fine-tuned, and on Design2Code the Moondream2 model trained with WAFFLE stays below GPT-4o and GPT-4o mini. The Gemini 1.5 Pro baseline answered 384 Design2Code instances. Human ranking used 50 samples per dataset and two annotators. Commercial-model prompting results are historical baselines, not current rankings.

Why it fits this collection

Adds a training method for screenshot-to-HTML website reconstruction that builds DOM hierarchy into attention, complementing the collection's screenshot-to-code benchmarks, datasets and prompting pipelines.

Publication and review details

Venue
ACL 2025
Status
Conference paper
Publication / posting date
July 2025
Date source
The official ACL Anthology record gives July 2025. Crossref records only the year; the July 27–August 1 conference dates printed in the PDF are not treated as a publication day.
First arXiv submission
October 24, 2024
arXiv
2410.18362
Reviewed arXiv version
v3
DOI
10.18653/v1/2025.acl-long.1208

Publication identity follows the ACL Anthology record and PDF. Reviewed arXiv v3 of 2026-03-02 (v1 2024-10-24, v2 2025-06-24) is titled “WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development”; the repository README and model card use “WAFFLE: Multi-Modal Model for Automated Front-End Development” and cite the arXiv preprint. The paper's availability statement links lt-asset/Waffle. The reviewed 2025-05-28 snapshot is BSD-3-Clause and provides VLM-WebSight inference, Selenium/Chrome rendering, the HTML tree-attention utilities, a contrastive dataset class reading a local load_from_disk dataset, and 500 WebSight-Test PNG screenshots without reference HTML. Although README describes a training file, the snapshot has no training script, Moondream2 code, metric implementation or mutated training set; no public copy of the 231,940-pair set was found in the lt-asset Hugging Face organization. Weights are on Hugging Face as lt-asset/Waffle_VLM_WebSight (7B). The GitHub README releases them under Apache-2.0 subject to the base model's terms, while the model card declares bsd-3-clause-clear. Documented dependencies include Python 3.10.14, PyTorch 2.3.0, Transformers 4.41.1, PEFT 0.11.1 and DeepSpeed 0.14.1; the inference script expects a CUDA GPU and has placeholder paths.

Sources reviewed

Reviewed against the paper and its primary sources. Cite the original authors using the linked paper or BibTeX.

Report an outdated entry Suggest a translation fix

Primary sources

  1. https://aclanthology.org/2025.acl-long.1208/
  2. https://aclanthology.org/2025.acl-long.1208.pdf
  3. https://api.crossref.org/works/10.18653/v1/2025.acl-long.1208
  4. https://arxiv.org/abs/2410.18362
  5. README.md GitHub · lt-asset/Waffle · Revision 5ca57f8
  6. LICENSE GitHub · lt-asset/Waffle · Revision 5ca57f8
  7. vlm_websight GitHub · lt-asset/Waffle · Revision 5ca57f8
  8. vlm_websight/eval_websight.py GitHub · lt-asset/Waffle · Revision 5ca57f8
  9. vlm_websight/dataset.py GitHub · lt-asset/Waffle · Revision 5ca57f8
  10. WebSight-Test GitHub · lt-asset/Waffle · Revision 5ca57f8
  11. https://huggingface.co/lt-asset/Waffle_VLM_WebSight/blob/296f6de57f8e646b1455ca00e3bc596ac0ea9429/README.md
  12. https://huggingface.co/HuggingFaceM4/VLM_WebSight_finetuned
↑↓ move↵ open⌘↵ new tab