What You Don't Know Can Hurt You: How Well do Latent Safety Filters Understand Partially Observable Safety Constraints?

TL;DR

The paper addresses safety in control under partial observability by analyzing latent safety filters that operate on learned latent states. It introduces a mutual information based measure to diagnose when safety-relevant features are observable and demonstrates that RGB only latent representations can yield myopic safety policies, while multimodal training with infrared signals shapes the latent space to be safety predictive even when deployment relies on RGB alone. The contributions include a formal MI-based observability diagnostic, a multimodal supervised training recipe that improves safety predictions without requiring extra modalities at deployment, and hardware validation on a Franka Panda manipulator to prevent wax overheating. This work advances robust safe control in vision-based robotics by bridging representation learning with uncertainty aware safety planning, and highlights practical limitations and avenues for future formal guarantees.

Abstract

Safe control techniques, such as Hamilton-Jacobi reachability, provide principled methods for synthesizing safety-preserving robot policies but typically assume hand-designed state spaces and full observability. Recent work has relaxed these assumptions via latent-space safe control, where state representations and dynamics are learned jointly through world models that reconstruct future high-dimensional observations (e.g., RGB images) from current observations and actions. This enables safety constraints that are difficult to specify analytically (e.g., spilling) to be framed as classification problems in latent space, allowing controllers to operate directly from raw observations. However, these methods assume that safety-critical features are observable in the learned latent state. We ask: when are latent state spaces sufficient for safe control? To study this, we examine temperature-based failures, comparable to overheating in cooking or manufacturing tasks, and find that RGB-only observations can produce myopic safety behaviors, e.g., avoiding seeing failure states rather than preventing failure itself. To predict such behaviors, we introduce a mutual information-based measure that identifies when observations fail to capture safety-relevant features. Finally, we propose a multimodal-supervised training strategy that shapes the latent state with additional sensory inputs during training, but requires no extra modalities at deployment, and validate our approach in simulation and on hardware with a Franka Research 3 manipulator preventing a pot of wax from overheating.

Figures (6)

• Figure 1: We design a series of controlled experiments to test how latent safety filters behave under partially observable constraints. Left: We find that safety filters that rely on latent state representations trained-on and deployed-with only RGB inputs behave unreliably when they must enforce constraints, such as temperature limits, that are not easily observable. Right: Training with rich, safety-relevant multimodal supervision shapes the latent state representation to enable safe control (e.g., lifting the pan before overheating), even when the robot is deployed with only RGB inputs at runtime.
• Figure 2: Experiment Testbeds. In simulation (left) and hardware (right) we control data from two sensors: RGB and infrared (IR) camera. In our controlled experiments, the ground-truth safety-relevant state variable is heat, which is more observable from the IR data than the RGB.
• Figure 3: Safety Controllers without & with Observability. In the Thermal Unicycle example, the robot only appears burnt in the RGB camera after it leaves the hot plate . Left: The safety policy is myopic when trained within a world model with an incomplete state representation: it stays in the hot region, avoiding seeing failure. Right: When the world model is trained with sufficiently rich observations (RGB + IR), the resulting safety controller makes the correct decision and accelerates out of the hot plate to prevent overheating.
• Figure 4: Real vs. Imagined Observations & Safety Labels.Open-loop prediction of an evaluation subtrajectory reveals that $\textbf{WM}_\textbf{RGB}$ incorrectly predicts the safety outcomes of actions, such as predicting that lifting the wax plate leads to failure. In contrast, observing temperature via IR images allows the $\textbf{WM}_\textbf{MM}$ to correctly classify that this action leads to a safe outcome.
• Figure 5: Distribution of ground-truth pixel intensities when the safety filter intervenes. The $\textbf{WM}_\textbf{MM}$ policy that takes in IR images concentrates just prior to the safety critical threshold used for labeling, whereas $\textbf{WM}_\textbf{RGB}$ and $\textbf{WM}_\textbf{RGB-MM}$ that take only RGB as input are spread more conservatively, indicating that the policy is unable to discern when to filter.