How Hard is it to Confuse a World Model?

TL;DR

The paper addresses how to quantify exploration difficulty under uncertain neural world dynamics by defining and computing the most confusing instance as a constrained optimization problem. It introduces suboptimality cost $K(s_0, \pi^*, \pi_{\text{sub}})$ as the infimum of the KL divergence over models that reverse policy rankings, solved via a Lagrangian relaxation that alternates between optimizing the perturbed dynamics and adjusting the constraint penalty. An empirical study using a VAE-based world dynamics model demonstrates that suboptimality cost increases with model quality, indicating that better-trained dynamics are less susceptible to confusion and thus provide stronger guidance for exploration. The findings offer a principled uncertainty metric for deep model-based RL and suggest directions for exploration strategies that prioritize decision-relevant uncertainty regions, with potential applicability across architectures and partially observable settings.

Abstract

In reinforcement learning (RL) theory, the concept of most confusing instances is central to establishing regret lower bounds, that is, the minimal exploration needed to solve a problem. Given a reference model and its optimal policy, a most confusing instance is the statistically closest alternative model that makes a suboptimal policy optimal. While this concept is well-studied in multi-armed bandits and ergodic tabular Markov decision processes, constructing such instances remains an open question in the general case. In this paper, we formalize this problem for neural network world models as a constrained optimization: finding a modified model that is statistically close to the reference one, while producing divergent performance between optimal and suboptimal policies. We propose an adversarial training procedure to solve this problem and conduct an empirical study across world models of varying quality. Our results suggest that the degree of achievable confusion correlates with uncertainty in the approximate model, which may inform theoretically-grounded exploration strategies for deep model-based RL.

Figures (4)

• Figure 1: Maze configurations for policy construction.Left: Original U-maze where $\pi^*$ follows the shortest path around the obstacle. Right: Modified maze where the wall configuration is altered, allowing $\pi_{\text{sub}}$ to take a direct vertical path. When executed in the original maze, $\pi_{\text{sub}}$ cannot pass through the wall and must take a longer route.
• Figure 2: Confusing instance search optimization dynamics.Left: KL divergence (logarithmic scale) and constraint evolution over training iterations. Right: Time to first constraint satisfaction.
• Figure 3: Statistical closeness of confusing instances.Left/Center: State visitation densities (log scale) under random policy. Right: Log-density ratio shows localized changes: blue indicates higher probability in the confusing instance, red indicates lower probability.
• Figure 4: Trajectory evolution during confusing instance search. Rows show reference models of increasing quality (epochs 100-500). Columns show optimization progress (iterations 0-1800). Green: optimal policy $\pi^*$. Red: suboptimal policy $\pi_{\text{sub}}$. Better-trained models resist confusion longer, with successful suboptimal trajectories appearing much later in the optimization.

Theorems & Definitions (2)

• Definition 1: Confusing Instance
• Definition 2: Suboptimality Cost