Fare: Failure Resilience in Learned Visual Navigation Control

TL;DR

The paper addresses failure-prone visual navigation under imitation learning by introducing Fare, a framework that embeds OOD detection and recognition directly into the policy and couples it with an informed recovery mechanism. It uses a Variational Information Bottleneck to learn task-relevant latent representations, applies conformal prediction for robust OOD detection, and uses Grad-CAM to localize OOD-relevant regions, enabling a lightweight recovery policy guided by spatial cues. The approach is validated on two diverse policies across real-world indoor/outdoor routes, demonstrating effective failure detection, recognition, and autonomous recovery without relying on failure data. The findings show that task-aware, in-policy OOD detection and recognition substantially improve robustness and enable long-range navigation in open-world environments with minimal external supervision. This has practical implications for deploying autonomous navigators in dynamic settings where failures are inevitable and human intervention should be minimized.

Abstract

While imitation learning (IL) enables effective visual navigation, IL policies are prone to unpredictable failures in out-of-distribution (OOD) scenarios. We advance the notion of failure-resilient policies, which not only detect failures but also recover from them automatically. Failure recognition that identifies the factors causing failure is key to informing recovery: e.g. pinpointing image regions triggering failure detections can provide cues to guide recovery. We present Fare, a framework to construct failure-resilient IL policies, embedding OOD-detection and recognition in them without using explicit failure data, and pairing them with recovery heuristics. Real-world experiments show that Fare enables failure recovery across two different policy architectures, enabling robust long-range navigation in complex environments.

Figures (6)

• Figure 1: Fare enables automatic recovery from policy failures. Limited training data causes failures on OOD scenarios, e.g. sensor failures, close dynamic obstacles etc.Fare not only detects these failures but also recognises failure causes in the input image, enabling informed recovery.
• Figure 2: Fare framework for failure resilience.Fare achieves failure-resilience through augmenting learned policies with OOD-awareness. Apart from generating actions, OOD-aware policies also detect when OOD inputs are encountered ($b{}_t$) and localise the OOD features in the observations ($M{}_t$). Detection and recognition enable informed recovery from OOD or failure scenarios. Fare employs a heuristic recovery policy taking $b{}_t, M{}_t$ as inputs.
• Figure 3: Comparison to baselines for OOD detection and recognition. (a) ROC curves for OOD scores for detection. (b-d): ROC curves based on score from summing pixels across the heatmap. We compute scores per heatmap bin: Left, Middle, Right.
• Figure 4: OOD detection and recognition examples. We visualise selected reconstruction and Fare methods. (a-c): Reconstruction-based methods struggle to detect failures like blocked pathways when there few visual features. (d-f): VAE-R indiscriminately highlights high-frequency features, falsely detecting failures. Fare-DEC is sensitive to obstacles (e.g. chairs, tables), prematurely triggering a detection. (g-l): VAE-KL offers smoother detections than VAE-R, but may not identify areas directly impacting navigation. In contrast, Fare-DEC correctly highlights the agent's leg (i) and the door (l) as failure causes.
• Figure 5: Failure recovery with Fare-GNM. (a): Robot is blocked by a pedestrian passing close by in front. It first backtracks (T=1s) to avoid the pedestrian, localises the pedestrian on its right (T=2s) and pivots in the opposite direction to recover (T=3s). (b): Robot is issued an infeasible goal behind the glass door. It perturbs locally to various directions, but finds itself always blocked by obstacles, thus terminating and seeking help.