The Impact of Element Ordering on LM Agent Performance

TL;DR

This work investigates how UI element ordering in an LM agent's input influences performance when only pixel-based GUI information is available. Using ablations on VisualWebArena and OmniACT, the authors show that element ordering is a dominant factor, with random orders severely reducing task success; text representations remain essential, and captions outperform alt text. They propose and evaluate ordering methods, notably 2D-to-1D dimensionality reduction via t-SNE, which consistently improves performance, especially for larger models and harder tasks, and they train a Faster-RCNN–based UI element detector to derive elements from pixels. Their end-to-end approach, combining pixel-based element detection with t-SNE ordering and a high-level action space, achieves state-of-the-art results on OmniACT, more than doubling previous performance. The work highlights a practical path to robust LM-enabled GUI navigation in environments lacking explicit DOM or accessibility structures, with broad implications for building adaptable, pixel-only agents.

Abstract

There has been a surge of interest in language model agents that can navigate virtual environments such as the web or desktop. To navigate such environments, agents benefit from information on the various elements (e.g., buttons, text, or images) present. It remains unclear which element attributes have the greatest impact on agent performance, especially in environments that only provide a graphical representation (i.e., pixels). Here we find that the ordering in which elements are presented to the language model is surprisingly impactful--randomizing element ordering in a webpage degrades agent performance comparably to removing all visible text from an agent's state representation. While a webpage provides a hierarchical ordering of elements, there is no such ordering when parsing elements directly from pixels. Moreover, as tasks become more challenging and models more sophisticated, our experiments suggest that the impact of ordering increases. Finding an effective ordering is non-trivial. We investigate the impact of various element ordering methods in web and desktop environments. We find that dimensionality reduction provides a viable ordering for pixel-only environments. We train a UI element detection model to derive elements from pixels and apply our findings to an agent benchmark--OmniACT--where we only have access to pixels. Our method completes more than two times as many tasks on average relative to the previous state-of-the-art.

Figures (3)

• Figure 1: We are motivated by the goal of enabling agents to act on environments where an underlying DOM does not exist. Instead, the agent must determine its next action using only the environment's graphical representations. In Step 1, we first detect a list of unordered UI elements using an object detection model and identify them with bounding boxes. In Step 2, we convert these UI elements to their text representation. In Step 3, we order the elements via 2D-to-1D dimensionality reduction. Due to the sequential nature of a language model, elements are always presented in a specific order to the LM Agent. Finding an effective ordering is non-trivial, yet can significantly affect agent performance. Elements that are visually close together are often functionally associated with each other. t-SNE's ability to retain local structure allows it to generate an effective ordering.
• Figure 2: Many software applications lack informative accessibility trees or DOMs. The accessibility tree for a popular game development engine (Godot, left) contains only the exit, minimize, and full screen buttons. For a presentation slide (Google Slides, right), no interactive elements (e.g. title and subtitle boxes) are present in the DOM. Language model agents rely on this information to navigate applications, and agent performance can accordingly be compromised in scenarios where it is incomplete.
• Figure 3: Differences in reward from random ordering. Element counts are divided based on the median. Across all datasets and element sources, the impact of a proper ordering method increases with element count.