Language-based Trial and Error Falls Behind in the Era of Experience

TL;DR

Language-based Trial and Error Falls Behind in the Era of Experience addresses the limitation of large language models when facing unseen, nonlinguistic environments due to expensive exploration and high-dimensional action spaces. The authors introduce SCOUT, a framework that delegates exploration to lightweight scouts to rapidly learn environmental dynamics, then distills these trajectories into the LLM via supervised fine-tuning before refining with multi-turn RL. Across six unseen symbolic and spatial tasks, including long-horizon challenges like 2048 and Rubik's Cube, SCOUT achieves an average score of 0.86 on Qwen2.5-3B-Instruct and significantly outperforms baselines while reducing GPU-hours by about 60%. This approach demonstrates that sub-scale collaboration can unlock latent world knowledge in LLMs, enabling scalable, resource-efficient language-based agents with strong generalization to unseen environments.

Abstract

While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT (Sub-Scale Collaboration On Unseen Tasks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption.

Figures (6)

• Figure 1: Exploration and Distillation Stage will either directly award the language models the skills, such as the performance on Sokoban-Box2 (below) that leads to direct convergence (0.0 to 0.45), or indirectly teach relevant knowledge that will be later activated via Evolving Stage, such as the performance on Sudoku (above figure, from 0.0, to 0.29, then to 0.97).
• Figure 2: Overview of the SCOUT framework. The pipeline consists of three stages: (1) Exploration Stage: Lightweight scouts efficiently capture environmental dynamics to generate expert trajectories; (2) Distillation Stage: These trajectories are textualized to "warm-up" the LLM via supervised fine-tuning; (3) Evolving Stage: The LLM further refines its reasoning and decision making capabilities through multi-turn PPO.
• Figure 3: Comparison of task performance during sequential RL. While the Sequential RL (Left) exhibits some performance degradation on previously learned tasks, SCOUT (Right) successfully preserves historical task knowledge (e.g., Bandit, FrozenLake) while adapting to new environments (e.g., Sudoku), achieving a near optimal multi-task agent.
• Figure 4: Scout-DQN detailed performance on 6 unseen tasks.
• Figure 5: Scout-PPO detailed performance on 6 unseen tasks.