DreamerV3-XP: Optimizing exploration through uncertainty estimation

TL;DR

DreamerV3-XP enhances DreamerV3 with two core improvements to boost data efficiency: a prioritized replay buffer that scores trajectories using return, reconstruction error, and value error, and an intrinsic reward based on ensemble disagreement over predicted rewards to drive exploration. The total reward blends extrinsic and intrinsic signals via $r_t^{\text{total}} = \lambda r_t^{\text{ext}} + (1-\lambda) r_t^{\text{intr}}$, while replay prioritization uses $s_i = (\lambda_r + \lambda_\delta \delta_i) R_i + \lambda_\epsilon \epsilon_i$. Empirical evaluation on Atari100k and DMC Vision shows faster learning and reduced dynamics model loss, particularly in sparse-reward tasks, with modest gains from latent reward disagreement but clear gains from optimized replay. The work demonstrates that epistemic uncertainty-driven exploration and informative experience replay can substantially improve sample efficiency in world-model-based RL, enabling faster policy learning with lower model loss.

Abstract

We introduce DreamerV3-XP, an extension of DreamerV3 that improves exploration and learning efficiency. This includes (i) a prioritized replay buffer, scoring trajectories by return, reconstruction loss, and value error and (ii) an intrinsic reward based on disagreement over predicted environment rewards from an ensemble of world models. DreamerV3-XP is evaluated on a subset of Atari100k and DeepMind Control Visual Benchmark tasks, confirming the original DreamerV3 results and showing that our extensions lead to faster learning and lower dynamics model loss, particularly in sparse-reward settings.

Figures (4)

• Figure 1: Reproduction results across five tasks. Mean episode return over 2 seeds is shown (with shaded variance), except for ⁎, which uses a single seed due to computational constraints.
• Figure 2: Mean (2 seeds) episode scores for Krull and Cup Catch with reward disagreement.
• Figure 3: Mean dynamics model training loss across five tasks, comparing optimized replay to the default setting. Results are averaged over 2 seeds, except for ⁎, which uses a single seed due to computational constraints.
• Figure 4: Mean (2 seeds) performance and loss metrics on Krull for optimized replay and default.