FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

TL;DR

FronTalk tackles the challenge of front-end code generation in a realistic, multi-turn, multimodal setting by introducing a benchmark that pairs textual and visual user intents with a robust agent-based evaluation framework. The dataset comprises 100 dialogues (1,000 turns) and 3,676 test cases derived from real websites, with GPT-4o-powered data generation and manual refinement to ensure verifiability. The study reveals two key challenges: a widespread forgetting issue where later turns overwrite earlier implementations, and difficulties in interpreting visual feedback, particularly for open-source vision-language models; it also introduces AceCoder, an agent-critique baseline that substantially reduces forgetting and improves pass rates, especially with textual feedback. Overall, FronTalk provides a solid foundation for advancing multi-turn, multi-modal front-end coding and offers a rigorous platform for evaluating interplay between instruction following and user experience in interactive web development.

Abstract

We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generation with multi-modal feedback. In front-end development, visual artifacts such as sketches, mockups and annotated creenshots are essential for conveying design intent, yet their role in multi-turn code generation remains largely unexplored. To address this gap, we focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues derived from real-world websites across diverse domains such as news, finance, and art. Each turn features both a textual instruction and an equivalent visual instruction, each representing the same user intent. To comprehensively evaluate model performance, we propose a novel agent-based evaluation framework leveraging a web agent to simulate users and explore the website, and thus measuring both functional correctness and user experience. Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature: (1) a significant forgetting issue where models overwrite previously implemented features, resulting in task failures, and (2) a persistent challenge in interpreting visual feedback, especially for open-source vision-language models (VLMs). We propose a strong baseline to tackle the forgetting issue with AceCoder, a method that critiques the implementation of every past instruction using an autonomous web agent. This approach significantly reduces forgetting to nearly zero and improves the performance by up to 9.3% (56.0% to 65.3%). Overall, we aim to provide a solid foundation for future research in front-end development and the general interaction dynamics of multi-turn, multi-modal code generation. Code and data are released at https://github.com/shirley-wu/frontalk

Figures (20)

• Figure 1: Focusing on the front-end development task, FronTalk benchmark explores a novel setting with multi-turn interactions and multi-modal user feedback.
• Figure 2: Website summary topics distribution.
• Figure 3: Illustration of the user simulators. Both simulators aim to contextualize the user intent based on the current website state: The textual simulator rewrites the textual content, while the visual simulator draws sketches and annotations on top of website screenshots.
• Figure 4: Error breakdown in interpreting visual instructions.
• Figure 5: Pass rate across different turns. We show the pass rate of intermediate outputs, i.e. PR$(o_t|\mathbf{i}_t)$, to factor out the forgetting issue.