Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration

Yiyuan Pan, Zhe Liu, Hesheng Wang

TL;DR

This work tackles autonomous exploration in sparse-reward, partially observable multi-agent reinforcement learning by introducing CERMIC, a modular framework that calibrates intrinsic curiosity with inferred multi-agent context. Grounded in the Information Bottleneck, CERMIC learns a contextualized exploration representation via a dynamic intention graph and produces theoretically grounded intrinsic rewards, while filtering noisy or socially irrelevant novelty. The method offers two key components: a novelty-driven exploration objective that optimizes $I(X_t;S_{t+1})$, and a robust, socially-aware exploitation objective that minimizes $I(X_t;[S_t,A_t])$ conditioned on inferred agent intentions, implemented through a graph-based memory and InfoNCE bounds. Empirically, CERMIC-enhanced agents achieve state-of-the-art performance across VMAS, MeltingPot, and SMACv2 in sparse-reward settings, demonstrating improved exploration stability and leveraging social context to improve coordination without centralized communication.

Abstract

Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers a powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit a uniform novelty bias, treating all unexpected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in decentralized, communication-free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observing peers, we propose a novel approach to enhance multi-agent exploration. We introduce CERMIC, a principled framework that empowers agents to robustly filter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC generates theoretically-grounded intrinsic rewards, encouraging agents to explore state transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse-reward environments.

Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration

TL;DR

This work tackles autonomous exploration in sparse-reward, partially observable multi-agent reinforcement learning by introducing CERMIC, a modular framework that calibrates intrinsic curiosity with inferred multi-agent context. Grounded in the Information Bottleneck, CERMIC learns a contextualized exploration representation via a dynamic intention graph and produces theoretically grounded intrinsic rewards, while filtering noisy or socially irrelevant novelty. The method offers two key components: a novelty-driven exploration objective that optimizes , and a robust, socially-aware exploitation objective that minimizes conditioned on inferred agent intentions, implemented through a graph-based memory and InfoNCE bounds. Empirically, CERMIC-enhanced agents achieve state-of-the-art performance across VMAS, MeltingPot, and SMACv2 in sparse-reward settings, demonstrating improved exploration stability and leveraging social context to improve coordination without centralized communication.

Abstract

Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers a powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit a uniform novelty bias, treating all unexpected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in decentralized, communication-free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observing peers, we propose a novel approach to enhance multi-agent exploration. We introduce CERMIC, a principled framework that empowers agents to robustly filter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC generates theoretically-grounded intrinsic rewards, encouraging agents to explore state transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse-reward environments.

Paper Structure

This paper contains 27 sections, 1 theorem, 48 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Method
  5. Novelty-Driven Exploration
  6. Multi-Agent Contextualized Exploitation
  7. Loss Module
  8. Intrinsic Reward Module
  9. Experiments
  10. Task Setup
  11. Comparison with State-of-the-Art
  12. Qualitative and Quantitative Analysis
  13. Ablation Study
  14. Generalization Across Reward Densities
  15. Conclusion
  16. ...and 12 more sections

Key Result

Theorem 1

Consider a linear MDP setting where the estimation noises of optimal parameters and transition dynamics are assumed to follow standard Gaussian distributions $\mathcal{N}(0, \mathbf{I})$. Then, for any tuning parameter $\rho > 0$, it holds that

Figures (6)

  • Figure 1: Workflow of Cermic. (Left) Illustrates Cermic's per-timestep objective. (Right) Depicts the network architecture. By projecting the raw observation $o_t$ into a lower-dimensional embedding $s_t$, subsequent computations within Cermic operate on compact representations, contributing to its overall computational efficiency.
  • Figure 2: Visualization of agent observation embeddings $s_t$ and latent states $x_t$. (Left) Comparative $s_t$ distributions for agents under Cermic-augmented vs. baseline algorithms. (Right) Influence of curiosity-driven latent states $x_t$ on task exploration.
  • Figure 3: The impact of pretrained intention memory module on Cermic's performances
  • Figure 4: Comparative performance on VMAS.
  • Figure 5: Performance comparison under environmental noise on VMAS task.
  • ...and 1 more figures

Theorems & Definitions (1)

  • Theorem 1