Experience-Driven Exploration for Efficient API-Free AI Agents

TL;DR

KG-Agent tackles the problem of open-ended, API-free GUI agents by structuring pixel-level interactions into a persistent State-Action Knowledge Graph that connects functionally similar states. A hybrid intrinsic reward, based on the SA-KG topology, balances exploitation of high-value paths with targeted exploration, enabling robust long-term planning. Empirical results in Slay the Spire and Civilization V show superior exploration efficiency and strategic depth, with high execution fidelity and reduced token costs. The approach advances API-free autonomy by enabling reuse and generalization of experiences across visually diverse but functionally related states, moving toward scalable, general-purpose AI agents in GUI-rich environments.

Abstract

Most existing software lacks accessible Application Programming Interfaces (APIs), requiring agents to operate solely through pixel-based Graphical User Interfaces (GUIs). In this API-free setting, large language model (LLM)-based agents face severe efficiency bottlenecks: limited to local visual experiences, they make myopic decisions and rely on inefficient trial-and-error, hindering both skill acquisition and long-term planning. To address these challenges, we propose KG-Agent, an experience-driven learning framework that structures an agent's raw pixel-level interactions into a persistent State-Action Knowledge Graph (SA-KG). KG-Agent overcomes inefficient exploration by linking functionally similar but visually distinct GUI states, forming a rich neighborhood of experience that enables the agent to generalize from a diverse set of historical strategies. To support long-horizon reasoning, we design a hybrid intrinsic reward mechanism based on the graph topology, combining a state value reward for exploiting known high-value pathways with a novelty reward that encourages targeted exploration. This approach decouples strategic planning from pure discovery, allowing the agent to effectively value setup actions with delayed gratification. We evaluate KG-Agent in two complex, open-ended GUI-based decision-making environments (Civilization V and Slay the Spire), demonstrating significant improvements in exploration efficiency and strategic depth over the state-of-the-art methods.

Figures (13)

• Figure 1: From Local Memory to Global Strategy. (a) A conventional API-free GUI-based agent relies on a simple skill database, where experiences are isolated. Its decision-making is a reactive trial-and-error process guided by myopic rewards like visual change, leading to inefficient exploration. (b) Our proposed KG-Agent leverages a structured knowledge graph as its skill library. This graph connects experiences, enabling a strategic "exploit or explore" mechanism, and the learning of API-free GUI-based agent is guided by an advanced implicit reward that captures long-term value, accelerating knowledge transfer and fostering strategic planning.
• Figure 2: Overview of the proposed KG-Agent. For a given state $i$, the agent queries its memory to find a neighborhood of experience ($\mathcal{N}_\mathcal{E}$), where state nodes are connected by similarity edges (blue line) or skill edges (red arrow). If candidate skills $\mathbb{S}_{i}$ are retrieved, they are executed to transition the environment to state $j$. In addition, a VLM is central to this agent's reasoning, performing skill invocation, refinement, augmentation, and evaluation, where the refinement is guided by a hybrid implicit reward composed of state value reward and novelty reward.
• Figure 3: Game progression measured by Civilization V's technology tree. Our framework outperforms all baselines, including those with task-related priors.
• Figure 4: Visual evolution of the SA-KG from Round 0 to Round 3. Skill associations (red) and similarity links (blue) illustrate structural refinement and relational reasoning across rounds.
• Figure 5: Skill Invocation and critical skills. (a) Our framework executes high-reward skills even with minimal visual change between states. (b) The top 10 skill edges reflect a consistent focus on critical long-term actions, aligning with the game’s core progression mechanics.