GuideWeb: A Benchmark for Automatic In-App Guide Generation on Real-World Web UIs

TL;DR

GuideWeb introduces the first benchmark for automatic in-app guide generation on real-world web UIs, decomposing the problem into selecting guide-worthy interactive elements and generating concise, user-aligned guidance grounded in visible page content. The proposed two-stage framework (guide target identification followed by element-grounded guide generation) is paired with a comprehensive evaluation protocol spanning target precision/recall, text quality (BLEU/ROUGE-L), and exact-field grounding. A dedicated GuideWeb Agent, augmented with a Shorter context-reduction mechanism, substantially outperforms strong baselines across all metrics, highlighting the need for task-specific modeling beyond generic LLM capabilities. Results demonstrate that while most baselines over-generate targets, the GuideWeb Agent achieves stronger precision and more accurate, user-centered guidance, underscoring its potential to ease automatic DAP deployment. Limitations include focus on main pages, offline training, and lack of online adaptation, suggesting future work toward multi-page workflows and interactive learning in deployed systems.

Abstract

Digital Adoption Platform (DAP) provide web-based overlays that deliver operation guidance and contextual hints to help users navigate complex websites. Although modern DAP tools enable non-experts to author such guidance, maintaining these guides remains labor-intensive because website layouts and functionalities evolve continuously, which requires repeated manual updates and re-annotation. In this work, we introduce \textbf{GuideWeb}, a new benchmark for automatic in-app guide generation on real-world web UIs. GuideWeb formulates the task as producing page-level guidance by selecting \textbf{guide target elements} grounded in the webpage and generating concise guide text aligned with user intent. We also propose a comprehensive evaluation suite that jointly measures the accuracy of guide target element selection and the quality of generated intents and guide texts. Experiments show that our proposed \textbf{GuideWeb Agent} achieves \textbf{30.79\%} accuracy in guide target element prediction, while obtaining BLEU scores of \textbf{44.94} for intent generation and \textbf{21.34} for guide-text generation. Existing baselines perform substantially worse, which highlights that automatic guide generation remains challenging and that further advances are necessary before such systems can be reliably deployed in real-world settings.

Figures (7)

• Figure 1: Illustration of a digital adoption platform (DAP) as an in-app overlay for unfamiliar web UIs. Top: without guidance, the user cannot confidently identify the correct guide target element to proceed. Bottom: with a DAP overlay, the webpage is augmented with contextual hints and step-by-step instructions, enabling the user to complete the action efficiently.
• Figure 2: Overview of GuideWeb. Given the main page of a real-world website, an LLM-based agent identifies guide targets, namely interactive UI elements whose usage may benefit from guidance, and generates corresponding guide text grounded in visible on-page content. The examples on the right illustrate both correct guides and typical failure cases, where the agent produces low-utility guidance for elements that are already self-explanatory.
• Figure 3: A single guide annotation in GuideWeb, containing intent, action type, guide text, and DOM grounding fields.
• Figure 4: Overview of the GuideWeb construction pipeline with LLM-assisted annotation and human verification.
• Figure 5: Overview of the GuideWeb Agent with the Shorter mechanism. The agent is trained on the GuideWeb benchmark to perform automatic in-app guide generation. During inference, the Shorter module reduces input length by removing irrelevant content, truncating long text, and compressing headers, enabling faster inference and reduced computational cost while preserving guide-relevant information.