Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions
Pedram Agand, Mo Chen
TL;DR
MoReBRAC tackles offline RL distributional shift by combining a dual-recurrent world model with a hierarchical uncertainty stack to generate and filter high-fidelity synthetic transitions. The framework pre-trains a robust simulator and then uses uncertainty-guided simulated exploration, enforcing manifold membership via a VAE ELBO, local stability, and epistemic confidence before replay updates. Key contributions include the uncertainty stack, the VAE as a geometric anchor, and a hybrid prioritized replay scheme that balances real and synthetic data, yielding strong gains on suboptimal/random data and robust performance overall. This approach enables safer, scalable offline policy optimization with practical relevance to safety-critical robotics and industrial settings.
Abstract
Offline Reinforcement Learning (ORL) holds immense promise for safety-critical domains like industrial robotics, where real-time environmental interaction is often prohibitive. A primary obstacle in ORL remains the distributional shift between the static dataset and the learned policy, which typically mandates high degrees of conservatism that can restrain potential policy improvements. We present MoReBRAC, a model-based framework that addresses this limitation through Uncertainty-Aware latent synthesis. Instead of relying solely on the fixed data, MoReBRAC utilizes a dual-recurrent world model to synthesize high-fidelity transitions that augment the training manifold. To ensure the reliability of this synthetic data, we implement a hierarchical uncertainty pipeline integrating Variational Autoencoder (VAE) manifold detection, model sensitivity analysis, and Monte Carlo (MC) dropout. This multi-layered filtering process guarantees that only transitions residing within high-confidence regions of the learned dynamics are utilized. Our results on D4RL Gym-MuJoCo benchmarks reveal significant performance gains, particularly in ``random'' and ``suboptimal'' data regimes. We further provide insights into the role of the VAE as a geometric anchor and discuss the distributional trade-offs encountered when learning from near-optimal datasets.