CaveAgent: Transforming LLMs into Stateful Runtime Operators

TL;DR

CaveAgent introduces Stateful Runtime Management by decoupling reasoning and execution into a dual-stream architecture: a lightweight semantic stream for planning and a persistent Python runtime for stateful execution. By injecting and persisting Python objects as an external memory, CaveAgent eliminates context drift, enables lossless data flow, and provides verifiable intermediate states, supporting complex, interdependent tasks and long-horizon reasoning. Empirical results on Tau^2-bench and BFCL show CaveAgent often surpasses JSON-based baselines, with notable gains in Retail tasks and strong performance for code-centric models; token-efficiency studies indicate substantial reductions in prompt tokens due to single, interdependent executions. The framework also enables data-intensive workflows with data query, analysis, and visualization tasks while maintaining high accuracy and low token usage, underscoring the practical impact of runtime-mediated state management for robust, scalable LLM-driven automation and potential extensions to multi-agent coordination.

Abstract

LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms. Traditional approaches rely on procedural JSON-based function calling, which often struggles with long-horizon tasks due to fragile multi-turn dependencies and context drift. In this paper, we present CaveAgent, a framework that transforms the paradigm from "LLM-as-Text-Generator" to "LLM-as-Runtime-Operator." We introduce a Dual-stream Context Architecture that decouples state management into a lightweight semantic stream for reasoning and a persistent, deterministic Python Runtime stream for execution. In addition to leveraging code generation to efficiently resolve interdependent sub-tasks (e.g., loops, conditionals) in a single step, we introduce \textit{Stateful Runtime Management} in CaveAgent. Distinct from existing code-based approaches that remain text-bound and lack the support for external object injection and retrieval, CaveAgent injects, manipulates, and retrieves complex Python objects (e.g., DataFrames, database connections) that persist across turns. This persistence mechanism acts as a high-fidelity external memory to eliminate context drift, avoid catastrophic forgetting, while ensuring that processed data flows losslessly to downstream applications. Comprehensive evaluations on Tau$^2$-bench, BFCL and various case studies across representative SOTA LLMs demonstrate CaveAgent's superiority. Specifically, our framework achieves a 10.5\% success rate improvement on retail tasks and reduces total token consumption by 28.4\% in multi-turn scenarios. On data-intensive tasks, direct variable storage and retrieval reduces token consumption by 59\%, allowing CaveAgent to handle large-scale data that causes context overflow failures in both JSON-based and Code-based agents.

Figures (14)

• Figure 1: Town Simulation: a toy example for Stateful Runtime-Mediated Multi-Agent Collaboration.
• Figure 2: Key Advantages of CaveAgent
• Figure 3: Evolution of Agentic Tool Use
• Figure 4: Framework Overview
• Figure 5: Performance comparison between CaveAgent and traditional JSON-based Function Calling across three scenarios, focusing on execution steps, success rates, prompt token and completion token consumptions. Here, the steps means the number of turns needed for task completion, prompt tokens refers to the cumulative input tokens sent to the model across all turns (including system prompts, conversation history, and tool results), and completion tokens refers to the cumulative output tokens generated by the model (including reasoning, function calls, or code generation). The Func Call in right figure represents the abbrevation of Function Calling.