Spark: Strategic Policy-Aware Exploration via Dynamic Branching for Long-Horizon Agentic Learning

TL;DR

Spark tackles the scarcity of high-quality trajectories in long-horizon agentic learning by introducing dynamic, policy-aware branching at pivotal decision points. It formulates agentic tasks as POMDPs and employs a four-stage Spark process—root initialization, autonomous branching guided by intrinsic <explore> signals, budget enforcement, and tree-based policy updates—to concentrate exploration where it yields the most value. Theoretical and empirical analyses show that branching at critical junctures amplifies decision coverage and reduces wasted computation, leading to higher success rates, better sample and token efficiency, and robust out-of-domain generalization across ALFWorld, ScienceWorld, and WebShop. These results suggest that autonomous strategic exploration, driven by intrinsic uncertainty signals rather than human priors, offers a scalable path to efficient, generalizable agentic learning in complex environments.

Abstract

Reinforcement learning has empowered large language models to act as intelligent agents, yet training them for long-horizon tasks remains challenging due to the scarcity of high-quality trajectories, especially under limited resources. Existing methods typically scale up rollout sizes and indiscriminately allocate computational resources among intermediate steps. Such attempts inherently waste substantial computation budget on trivial steps while failing to guarantee sample quality. To address this, we propose \textbf{Spark} (\textbf{S}trategic \textbf{P}olicy-\textbf{A}ware explo\textbf{R}ation via \textbf{K}ey-state dynamic branching), a novel framework that selectively branches at critical decision states for resource-efficient exploration. Our key insight is to activate adaptive branching exploration at critical decision points to probe promising trajectories, thereby achieving precise resource allocation that prioritizes sampling quality over blind coverage. This design leverages the agent's intrinsic decision-making signals to reduce dependence on human priors, enabling the agent to autonomously expand exploration and achieve stronger generalization. Experiments across diverse tasks (e.g., embodied planning), demonstrate that \textsc{Spark} achieves superior success rates with significantly fewer training samples, exhibiting robust generalization even in unseen scenarios.

Figures (13)

• Figure 1: Paradigm comparison: uniform vs. strategic exploration. Standard RL (Left) wastes budget via uniform, independent sampling. In contrast, Spark (Right) employs dynamic branching at critical junctures for precise resource allocation, yielding higher-quality trajectories under the similar computational budget.
• Figure 2: Multi-benchmark performance comparison. SPARK outperforms all baselines across ALFWorld (L0-L2), ScienceWorld (L0-L2), and WebShop tasks, achieving +73.5% average improvement.
• Figure 3: Overview of Spark framework.Spark performs dynamic branching exploration via: (1) Root Initialization: diverse starting trajectories; (2) Autonomous Branching: selective expansion at high-uncertainty states using intrinsic <explore> signals; (3) Budget Enforcement: constraining tree growth within computational limits. The resulting trajectory trees are then used for Tree-Based Policy Optimization.
• Figure 4: Sample efficiency comparison.Spark surpasses GRPO@100% using only 20% data, while baselines often collapse in low-data regimes.
• Figure 5: Ablation on initial root count ($M$). We report success rates (%) with fixed total budget $N=8$.

Theorems & Definitions (2)

• proof : Heuristic Argument
• proof : Heuristic Argument