Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents

TL;DR

This work tackles the core problem of scaling LLM-based web agents by serializing web application state via DOM snapshots, which preserve structural UI information but can be prohibitively large. It introduces D2Snap, a DOM downsampling algorithm that conserves UI features across node types (container, content, interactive) and supports adaptive configurations using TextRank for text, semantic thresholds for attributes, and hierarchical merging. Empirical evaluation on the Online-Mind2Web dataset shows that D2Snap can achieve performance on par with or better than grounded GUI baselines (up to 73% success in best configurations) while reducing input size to within typical model context windows; hierarchy emerges as a key UI signal for LLMs. The study also finds limited benefit from vision inputs and highlights dataset size and cross-origin constraints as important considerations, outlining future work to blend DOM snapshots with scoped visuals and to broaden evaluation across more web tasks.

Abstract

Frontier LLMs only recently enabled serviceable, autonomous web agents. At that, a model poses as an instantaneous domain model backend. Ought to suggest interaction, it is consulted with a web-based task and respective application state. The key problem lies in application state serialisation - referred to as snapshot. State-of-the-art web agents are premised on grounded GUI snapshots, i.e., screenshots enhanced with visual cues. Not least to resemble human perception, but for images representing relatively cheap means of model input. LLM vision still lag behind code interpretation capabilities. DOM snapshots, which structurally resemble HTML, impose a desired alternative. Vast model input token size, however, disables reliable implementation with web agents to date. We propose D2Snap, a first-of-its-kind DOM downsampling algorithm. Based on a GPT-4o backend, we evaluate D2Snap on tasks sampled from the Online-Mind2Web dataset. The success rate of D2Snap-downsampled DOM snapshots (67%) matches a grounded GUI snapshot baseline (65%) - within the same input token order of magnitude (1e3). Our best evaluated configurations - one token order above, but within the model's context window - outperform this baseline by 8%. Our evaluation, moreover, yields that DOM-inherent hierarchy embodies a strong UI feature for LLMs.

Figures (5)

• Figure 1: Success rate per subject evaluated across the dataset. The rate of grounded GUI snapshots (GUI$_{gr.}$) represents the baseline. Except for adaptive configurations, all D2Snap subjects outperform the baseline. Limited at five iterations, AdaptiveD2Snap is able to downsample about two thirds of DOMs in the dataset for a strict token limit of 8,192. With a limit of 32,768, it is able to downsample without error. Hierarchy supports highest utility across the assessed UI features.
• Figure 2: Comparison of mean input size across subjects, and estimated token and byte size per subject snapshots. Both token and byte size strongly correlate for text-modal D2Snap subjects. For the baseline, grounded GUI snapshots (GUI$_{gr.}$), byte size scales way beyond remotely processed token size (hidden costs).
• Figure 3: Estimated input token size per snapshot created by the subject D2Snap$_{.6, .9, .3}$ across the entire dataset, sorted in ascending order. Greater part of snapshots meets the token order of the grounded GUI snapshots baseline. Only about a fifth of DOMs scale beyond an order of $1\mathrm{e}{4}$.
• Figure :
• Figure :