Nightmare Dreamer: Dreaming About Unsafe States And Planning Ahead
Oluwatosin Oseni, Shengjie Wang, Jun Zhu, Micah Corah
TL;DR
The paper tackles the safety guarantees gap in reinforcement learning for real-world tasks by formulating Safe RL as a CMDP and introducing Nightmare Dreamer, a model-based approach that uses a learned Recurrent State-Space Model to forecast future safety violations. It employs a bi-actor architecture with a Control actor for rewards and a Safe actor for constraints, combined with online planning that imagines rollouts and switches policies based on predicted costs; planning is augmented by a discriminator-based regularization to stabilize training. Key contributions include the dual-actor design, proactive safety planning via world-model rollouts, and discriminator-based policy regularization, which together yield near-zero constraint violations and significantly improved sample efficiency on Vision-based Safety Gymnasium Circle tasks (up to ~20x). This approach advances practical SafeRL by enabling proactive risk management and more efficient learning, with potential extensions to more complex environments and real robotic systems.
Abstract
Reinforcement Learning (RL) has shown remarkable success in real-world applications, particularly in robotics control. However, RL adoption remains limited due to insufficient safety guarantees. We introduce Nightmare Dreamer, a model-based Safe RL algorithm that addresses safety concerns by leveraging a learned world model to predict potential safety violations and plan actions accordingly. Nightmare Dreamer achieves nearly zero safety violations while maximizing rewards. Nightmare Dreamer outperforms model-free baselines on Safety Gymnasium tasks using only image observations, achieving nearly a 20x improvement in efficiency.