World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference

TL;DR

The paper addresses safety for learning-based controllers by using world-models during inference to monitor forecasted state trajectories against actual observations, triggering safety actions when discrepancies exceed thresholds. It extends DreamerV3-style world models to inference-time anomaly detection, confirming that both global and local environmental or actuator changes induce measurable prediction-errors across simulated and real robotic platforms. Key contributions include a horizon-based error formulation, visualization of prediction gaps in image space, and demonstrations of sim2real applicability that aid debugging and interpretability. The work suggests that, while not guaranteeing safety, inference-time world-model discrepancies offer a universal, task-agnostic mechanism to detect unfamiliar situations and guide corrective actions in real-world deployments.

Abstract

Learning-based controllers are often purposefully kept out of real-world applications due to concerns about their safety and reliability. We explore how state-of-the-art world models in Model-Based Reinforcement Learning can be utilized beyond the training phase to ensure a deployed policy only operates within regions of the state-space it is sufficiently familiar with. This is achieved by continuously monitoring discrepancies between a world model's predictions and observed system behavior during inference. It allows for triggering appropriate measures, such as an emergency stop, once an error threshold is surpassed. This does not require any task-specific knowledge and is thus universally applicable. Simulated experiments on established robot control tasks show the effectiveness of this method, recognizing changes in local robot geometry and global gravitational magnitude. Real-world experiments using an agile quadcopter further demonstrate the benefits of this approach by detecting unexpected forces acting on the vehicle. These results indicate how even in new and adverse conditions, safe and reliable operation of otherwise unpredictable learning-based controllers can be achieved.

Figures (5)

• Figure 1: Illustration of our proposed method. Instead of discarding a world model after training, it can be used during inference to continuously generate predictions into the future. These can then be compared to the real outcomes and thus provide a measure about the familiarity of the agent with the situation at hand. This, in turn can be used to implement universal safety checks for learning-based controllers.
• Figure 2: DMC's Walker-walk task with DreamerV3's world model's predictions (green) and actual measurements (blue), along with corresponding smoothed observation and reward error. After $500$ timesteps, gravity is increased by $50\%$.
• Figure 3: DMC's Manipulator-bring-ball task with DreamerV3's world model's predictions (green) and actual measurements (blue), along with corresponding smoothed observation and reward error. At timestep $480$ the gear ratio of the elbow joint is increased by a factor of three.
• Figure 4: DreamerV3's world model with visual input reacting to an unexpected force applied to the spacecraft in LunarLander-v2 after the third image, pushing it to the top-right. Both input and prediction images are averaged over $16$ timesteps to visualize movement. The resulting error heatmap displays the location of unexpected events and their magnitude. For visualization purposes, as well as noise reduction, we render multiple images into one here.
• Figure 5: The same experiment performed in simulation and the real-world. A quadcopter successively navigates the corners of a $1$ m2 square. At $240$ and $205$ timesteps, respectively, a sudden force briefly acts upon it. While world model predictions (color) and true measurements (color) closely align throughout the rest of the flight, this event produces a significant error.