Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Scalar Rewards to Potential Trends: Shaping Potential Landscapes for Model-Based Reinforcement Learning

Yao-Hui Li, Zeyu Wang, Xin Li, Wei Pang, Yingfang Yuan, Zhengkun Chen, Boya Zhang, Riashat Islam, Alex Lamb, Yonggang Zhang

TL;DR

This work tackles the challenge of sparse rewards in model-based reinforcement learning by shifting reward modeling from scalar regression to shaping an optimistic potential landscape. Building on PBRS, SLOPE defines a dynamic potential Φ(s) and constructs a reshaped reward $ ilde{r}$ to provide dense planning signals, while an optimism-driven distributional learning objective emphasizes upper quantiles to amplify rare successes. The authors prove convergence guarantees under a contraction condition and demonstrate substantial empirical gains across 30+ tasks on 5 benchmarks, including real-robot manipulation, by integrating with backbones like TD-MPC2 and DreamerV3. The approach yields faster learning, higher success rates, and robust performance with sparse feedback, suggesting broad applicability to real-world autonomous systems where dense reward engineering is impractical. In short, SLOPE offers a theoretically sound and practically effective route to enable reliable planning and learning under sparse rewards through optimistic potential landscapes.

Abstract

Model-based reinforcement learning (MBRL) achieves high sample efficiency by simulating future trajectories with learned dynamics and reward models. However, its effectiveness is severely compromised in sparse reward settings. The core limitation lies in the standard paradigm of regressing ground-truth scalar rewards: in sparse environments, this yields a flat, gradient-free landscape that fails to provide directional guidance for planning. To address this challenge, we propose Shaping Landscapes with Optimistic Potential Estimates (SLOPE), a novel framework that shifts reward modeling from predicting scalars to constructing informative potential landscapes. SLOPE employs optimistic distributional regression to estimate high-confidence upper bounds, which amplifies rare success signals and ensures sufficient exploration gradients. Evaluations on 30+ tasks across 5 benchmarks demonstrate that SLOPE consistently outperforms leading baselines in fully sparse, semi-sparse, and dense rewards.

From Scalar Rewards to Potential Trends: Shaping Potential Landscapes for Model-Based Reinforcement Learning

TL;DR

This work tackles the challenge of sparse rewards in model-based reinforcement learning by shifting reward modeling from scalar regression to shaping an optimistic potential landscape. Building on PBRS, SLOPE defines a dynamic potential Φ(s) and constructs a reshaped reward to provide dense planning signals, while an optimism-driven distributional learning objective emphasizes upper quantiles to amplify rare successes. The authors prove convergence guarantees under a contraction condition and demonstrate substantial empirical gains across 30+ tasks on 5 benchmarks, including real-robot manipulation, by integrating with backbones like TD-MPC2 and DreamerV3. The approach yields faster learning, higher success rates, and robust performance with sparse feedback, suggesting broad applicability to real-world autonomous systems where dense reward engineering is impractical. In short, SLOPE offers a theoretically sound and practically effective route to enable reliable planning and learning under sparse rewards through optimistic potential landscapes.

Abstract

Model-based reinforcement learning (MBRL) achieves high sample efficiency by simulating future trajectories with learned dynamics and reward models. However, its effectiveness is severely compromised in sparse reward settings. The core limitation lies in the standard paradigm of regressing ground-truth scalar rewards: in sparse environments, this yields a flat, gradient-free landscape that fails to provide directional guidance for planning. To address this challenge, we propose Shaping Landscapes with Optimistic Potential Estimates (SLOPE), a novel framework that shifts reward modeling from predicting scalars to constructing informative potential landscapes. SLOPE employs optimistic distributional regression to estimate high-confidence upper bounds, which amplifies rare success signals and ensures sufficient exploration gradients. Evaluations on 30+ tasks across 5 benchmarks demonstrate that SLOPE consistently outperforms leading baselines in fully sparse, semi-sparse, and dense rewards.
Paper Structure (57 sections, 3 theorems, 20 equations, 20 figures, 7 tables, 2 algorithms)

This paper contains 57 sections, 3 theorems, 20 equations, 20 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. MBRL with Sparse-Reward
  4. Reward Shaping
  5. Preliminaries
  6. Problem Formulation
  7. TD-MPC2
  8. The Challenge of Sparse Rewards.
  9. Methodology
  10. Model-Based Planning via Potential Landscape
  11. Optimism-Driven Landscape Shaping
  12. Accelerating MBRL with Sparse Rewards
  13. Experiments
  14. Experimental Settings
  15. Baselines.
  16. ...and 42 more sections

Key Result

Theorem 4.1

Given the MDP $\mathcal{M}$ with optimal policy $\pi^*$, the reshaped MDP $\widetilde{\mathcal{M}}$ with the reward function defined as: preserves the optimal policy, i.e., $\widetilde{\pi}^*= \pi^*$, for any bounded potential function $\Phi \colon \mathcal{S} \to \mathbb{R}$. Furthermore, the optimal value functions satisfy:

Figures (20)

  • Figure 1: Key challenges in sparse-reward MBRL. Left: Reward Learning under Data Imbalance. The scarcity of successful samples hinders the model from capturing valid reward patterns due to dataset imbalance. Right: Uninformative Planning. Fitting sparse scalars creates a gradient-free landscape, depriving the planner of directional guidance toward the goal.
  • Figure 2: Training framework of our method. Building upon MoDem’s multi-phase accelerated learning framework, we also introduce two training enhancements: (i) initializing MPPI sampling distribution from the prior policy $\pi_{\text{Prior}}$, and (ii) augmenting the demonstration buffer with successful trajectories. The shaped reward $\widetilde{r}$ is used for training both the reward model $R_{\theta}$ and the $Q$ value function $Q_{\theta}$. The environment input consists of multimodal observations $\mathbf{o}=(\mathbf{x,q})$, where $\mathbf{x}$ denotes raw RGB images captured by the robot’s camera, and $\mathbf{q}$ represents the robot’s proprioceptive sensory information, e.g., the gripper state. Subsequent observations are encoded by the target encoder (illustrated in grey) which is updated via an exponential moving average (EMA) of the online encoder.
  • Figure 3: Toy example on a $10 \times 10$ GridWorld. Left: The original environment with sparse rewards ($+1$ at goal, $0$ elsewhere). Right: The dense reward signal generated by PBRS using converged optimal value function $V^*$. $\blacksquare$: wall.
  • Figure 4: Average success rates across 20 tasks from 4 benchmarks. Curves and shaded areas represent the mean and 95% confidence intervals (CIs) over 5 independent runs. We include both sparse and semi-sparse tasks for ManiSkill3 and Meta-World, while RoboSuite and Adroit involve sparse rewards only, following their native task designs. See Appendix \ref{['sec:detail_results']} for individual task details.
  • Figure 5: Visualizations of representative tasks from 4 benchmarks used in our experiments: ManiSkill3, Meta-World, Robosuite and Adroit. For brevity, we use 'ms' to represent ManiSkill3 and 'mw' to represent Meta-World.
  • ...and 15 more figures

Theorems & Definitions (6)

  • Theorem 4.1: Potential-Based Reward Shapingng_shaping
  • Proposition 4.2: Contraction and Convergence
  • Remark 4.3
  • Lemma 1.1: Non-expansiveness of the Max Operator
  • proof
  • proof