CI4A: Semantic Component Interfaces for Agents Empowering Web Automation

TL;DR

This work introduces CI4A, a protocol that semantically encapsulates UI components into standardized tool primitives ⟨S, T, M⟩ to align agent reasoning with web interactions. Implemented as AntDX, CI4A instruments 23 Ant Design components to expose semantic state, executable tools, and metadata, enabling a hybrid action space agent (Eous) that combines high-level tools with low-level fallbacks. On a semantically refactored WebArena, Eous achieves SoTA task success (86.3%) and drastically reduces interaction steps, outperforming unimodal and multimodal baselines. The approach promises a scalable, agent-centric Web interface paradigm akin to an 'ARIA for Agents', with future work on automated instrumentation and online adaptation for open-world websites.

Abstract

While Large Language Models demonstrate remarkable proficiency in high-level semantic planning, they remain limited in handling fine-grained, low-level web component manipulations. To address this limitation, extensive research has focused on enhancing model grounding capabilities through techniques such as Reinforcement Learning. However, rather than compelling agents to adapt to human-centric interfaces, we propose constructing interaction interfaces specifically optimized for agents. This paper introduces Component Interface for Agent (CI4A), a semantic encapsulation mechanism that abstracts the complex interaction logic of UI components into a set of unified tool primitives accessible to agents. We implemented CI4A within Ant Design, an industrial-grade front-end framework, covering 23 categories of commonly used UI components. Furthermore, we developed a hybrid agent featuring an action space that dynamically updates according to the page state, enabling flexible invocation of available CI4A tools. Leveraging the CI4A-integrated Ant Design, we refactored and upgraded the WebArena benchmark to evaluate existing SoTA methods. Experimental results demonstrate that the CI4A-based agent significantly outperforms existing approaches, achieving a new SoTA task success rate of 86.3%, alongside substantial improvements in execution efficiency.

Figures (5)

• Figure 1: Fragile Atomic Operations vs. Robust CI4A. In contrast to traditional methods entangled in error-prone multi-step chains for date selection, CI4A collapses complex logic into deterministic single-step invocations, eliminating the uncertainty of long-sequence execution.
• Figure 2: CI4A’s Semantic State View ensures accurate state perception for hidden lazy-loaded data and ambiguous DOM states.
• Figure 3: CI4A workflow. Components register with a global registry, allowing agents to retrieve the triplet $\langle \mathcal{S_\mathcal{K}}, \Sigma_{\mathcal{T_\mathcal{K}}}, \mathcal{M_\mathcal{K}} \rangle$ for hybrid view construction. The agent then derives tools invocations and parameters from this view and executes tools via the registry.
• Figure 4: Action space distribution of Eous vs. Baselines. Compared to baselines that rely on lengthy low-level atomic actions, Eous leverages high-level semantic tools via CI4A to significantly collapse physical interaction chains, thereby reducing total steps.
• Figure 5: Performance across task lengths. Unlike baselines that degrade rapidly in long tasks due to error accumulation, Eous maintains high success rates, demonstrating superior robustness in long-horizon interactions.