RAPT: Model-Predictive Out-of-Distribution Detection and Failure Diagnosis for Sim-to-Real Humanoid Robots

TL;DR

RAPT addresses the challenge of silent, high-frequency out-of-distribution failures when transferring humanoid policies from simulation to the real world. It combines a lightweight probabilistic spatio-temporal model trained on nominal data with calibrated per-dimension NLL thresholds and a three-gate anomaly detector to enable reliable online detection at $50$ Hz. A post-hoc diagnostic pipeline links gradient-based temporal saliency to an LLM for zero-shot semantic failure classification, providing actionable root-cause insights with proprioceptive data. Across large-scale simulations and 24 real-world Unitree G1 deployments, RAPT outperforms baselines on detection metrics and delivers interpretable diagnostics, demonstrating practical impact for sim-to-real humanoid control.

Abstract

Deploying learned control policies on humanoid robots is challenging: policies that appear robust in simulation can execute confidently in out-of-distribution (OOD) states after Sim-to-Real transfer, leading to silent failures that risk hardware damage. Although anomaly detection can mitigate these failures, prior methods are often incompatible with high-rate control, poorly calibrated at the extremely low false-positive rates required for practical deployment, or operate as black boxes that provide a binary stop signal without explaining why the robot drifted from nominal behavior. We present RAPT, a lightweight, self-supervised deployment-time monitor for 50Hz humanoid control. RAPT learns a probabilistic spatio-temporal manifold of nominal execution from simulation and evaluates execution-time predictive deviation as a calibrated, per-dimension signal. This yields (i) reliable online OOD detection under strict false-positive constraints and (ii) a continuous, interpretable measure of Sim-to-Real mismatch that can be tracked over time to quantify how far deployment has drifted from training. Beyond detection, we introduce an automated post-hoc root-cause analysis pipeline that combines gradient-based temporal saliency derived from RAPT's reconstruction objective with LLM-based reasoning conditioned on saliency and joint kinematics to produce semantic failure diagnoses in a zero-shot setting. We evaluate RAPT on a Unitree G1 humanoid across four complex tasks in simulation and on physical hardware. In large-scale simulation, RAPT improves True Positive Rate (TPR) by 37% over the strongest baseline at a fixed episode-level false positive rate of 0.5%. On real-world deployments, RAPT achieves a 12.5% TPR improvement and provides actionable interpretability, reaching 75% root-cause classification accuracy across 16 real-world failures using only proprioceptive data.

Figures (3)

• Figure 1: RAPT overview. Real-world out-of-distribution (OOD) scenarios during humanoid deployment. RAPT detects anomalies online and triggers predefined safety responses, including safe stopping, controlled falling, and recovery behaviors.
• Figure 2: RAPT Method Overview: (A) RAPT OOD-detection architecture. (B) Hierarchical OOD pipeline using three statistical gates for real-time ($\sim$1.6ms) monitoring. (C) Detected anomalies trigger gradient-based saliency generation ($s_t$) for zero-shot diagnosis via a multi-modal LLM.
• Figure 3: RAPT on a real-world anomaly. Top: stumble on deformable ground during a low-magnitude walking command. Second: NLL margin to threshold, spiking at failure. Third: Top-5 temporal saliency heatmap. Bottom: LLM-based diagnosis from proprioceptive saliency and joint states.