SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation

TL;DR

SimuDICE is introduced, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model that achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.

Abstract

In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.

Figures (6)

• Figure 1: The components of SimuDICE and their interactions. Transitions adapted from Dyna-Q sutton1991dyna are in blue, while those unique to SimuDICE are depicted in black.
• Figure 2: Illustration of the suite of tasks considered in this work. These tasks require the RL agent to learn to navigate grid environments to accomplish certain tasks.
• Figure 3: Comparison of algorithm performance in discrete tabular environments: Taxi, CliffWalking, and FrozenLake, under varying epsilon-greedy data collection policies ($\epsilon = 0.1, 0.4, 0.7$). Each plot shows the average per-step reward as a function of the number of trajectories in the offline data. The results are averaged over 500 episodes and 5 different random seeds. The shaded regions represent the variance across the different seeds. PS represents the number of planning steps.
• Figure 4: Impact of the number of planning steps on the average per-step reward under different epsilon-greedy data collection policies, with varying epsilon values.
• Figure 5: Comparison of average per-step rewards achieved by SimuDICE using different sampling formulas, across different epsilon-greedy offline dataset collection policies.