Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology

Jian Xiong, Jingbo Zhou, Zihan Zhou, Yixiong Xiao, Le Zhang, Jingyong Ye, Rui Qian, Yang Zhou, Dejing Dou

TL;DR

This work demonstrates Tolman-style latent learning in large language models by introducing an unrewarded exploration phase before reward-based fine-tuning. Using Group Relative Policy Optimization with KL regularization and ratio clipping, the authors show that LLMs can improve internal representations and performance without rewards, and that later rewards yield additional gains—sometimes surpassing reward-only training. They provide theoretical guarantees for monotone improvements in latent utility under both discrete and continuous output spaces, supported by empirical results on mathematical reasoning and GUI agent tasks across multiple model families. The findings suggest a two-stage training paradigm may enhance generalization and flexibility in AI systems, bridging cognitive psychology insights with modern reinforcement fine-tuning while highlighting avenues for scaling and broader task coverage.

Abstract

Latent learning, classically theorized by Tolman, shows that biological agents (e.g., rats) can acquire internal representations of their environment without rewards, enabling rapid adaptation once rewards are introduced. In contrast, from a cognitive science perspective, reward learning remains overly dependent on external feedback, limiting flexibility and generalization. Although recent advances in the reasoning capabilities of large language models (LLMs), such as OpenAI-o1 and DeepSeek-R1, mark a significant breakthrough, these models still rely primarily on reward-centric reinforcement learning paradigms. Whether and how the well-established phenomenon of latent learning in psychology can inform or emerge within LLMs' training remains largely unexplored. In this work, we present novel findings from our experiments that LLMs also exhibit the latent learning dynamics. During an initial phase of unrewarded exploration, LLMs display modest performance improvements, as this phase allows LLMs to organize task-relevant knowledge without being constrained by reward-driven biases, and performance is further enhanced once rewards are introduced. LLMs post-trained under this two-stage exploration regime ultimately achieve higher competence than those post-trained with reward-based reinforcement learning throughout. Beyond these empirical observations, we also provide theoretical analyses for our experiments explaining why unrewarded exploration yields performance gains, offering a mechanistic account of these dynamics. Specifically, we conducted extensive experiments across multiple model families and diverse task domains to establish the existence of the latent learning dynamics in LLMs.

Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology

TL;DR

This work demonstrates Tolman-style latent learning in large language models by introducing an unrewarded exploration phase before reward-based fine-tuning. Using Group Relative Policy Optimization with KL regularization and ratio clipping, the authors show that LLMs can improve internal representations and performance without rewards, and that later rewards yield additional gains—sometimes surpassing reward-only training. They provide theoretical guarantees for monotone improvements in latent utility under both discrete and continuous output spaces, supported by empirical results on mathematical reasoning and GUI agent tasks across multiple model families. The findings suggest a two-stage training paradigm may enhance generalization and flexibility in AI systems, bridging cognitive psychology insights with modern reinforcement fine-tuning while highlighting avenues for scaling and broader task coverage.

Abstract

Latent learning, classically theorized by Tolman, shows that biological agents (e.g., rats) can acquire internal representations of their environment without rewards, enabling rapid adaptation once rewards are introduced. In contrast, from a cognitive science perspective, reward learning remains overly dependent on external feedback, limiting flexibility and generalization. Although recent advances in the reasoning capabilities of large language models (LLMs), such as OpenAI-o1 and DeepSeek-R1, mark a significant breakthrough, these models still rely primarily on reward-centric reinforcement learning paradigms. Whether and how the well-established phenomenon of latent learning in psychology can inform or emerge within LLMs' training remains largely unexplored. In this work, we present novel findings from our experiments that LLMs also exhibit the latent learning dynamics. During an initial phase of unrewarded exploration, LLMs display modest performance improvements, as this phase allows LLMs to organize task-relevant knowledge without being constrained by reward-driven biases, and performance is further enhanced once rewards are introduced. LLMs post-trained under this two-stage exploration regime ultimately achieve higher competence than those post-trained with reward-based reinforcement learning throughout. Beyond these empirical observations, we also provide theoretical analyses for our experiments explaining why unrewarded exploration yields performance gains, offering a mechanistic account of these dynamics. Specifically, we conducted extensive experiments across multiple model families and diverse task domains to establish the existence of the latent learning dynamics in LLMs.
Paper Structure (24 sections, 2 theorems, 69 equations, 1 figure, 6 tables)

This paper contains 24 sections, 2 theorems, 69 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Latent Learning in Psychology
  4. Reinforcement Learning Paradigm for Large Language Models
  5. Bridging Psychology and Artificial Intelligence
  6. Method
  7. Rewarded Training
  8. Unrewarded Exploration
  9. Theoretical Analyses
  10. Experiments
  11. Models for Experiments
  12. Datasets for experiments
  13. Mathematical reasoning
  14. GUI agent tasks
  15. Reward rules
  16. ...and 9 more sections

Key Result

Theorem 1

Consider an autoregressive language model policy $\pi_\theta$ that defines, for each input prompt $x$ and token prefix $y_{<t}$, a conditional distribution$\pi_\theta(y_t \mid x,y_{<t})$ over the next token $y_t$. The model is trained with Group Relative Policy Optimization where the per-token advan where $\tau(x,y_{<t})>0$ is the unique normalizing constant satisfying $\sum_{y_t \in \mathcal{V}}

Figures (1)

  • Figure 1: Overview of tasks and models we use in this work, and training paradigm. A: Examples for the different tasks. Experiment tasks include mathematical reasoning and GUI agent task. B: Used LLMs include different series and parameter sizes. C: Our training process. For every task, questions are given to LLMs to generate a group of responses. The green arrow represents learning without rewards. The red arrow represents learning with rewards.

Theorems & Definitions (5)

  • Definition 1: Probability Simplex
  • Theorem 1: Performance improvement without rewards in sparse setting
  • Theorem 2: Performance improvement without rewards in continuous setting
  • proof
  • proof