Do You Know the Way? Human-in-the-Loop Understanding for Fast Traversability Estimation in Mobile Robotics

TL;DR

This paper introduces CHUNGUS, a human-in-the-loop traversability learning framework that enables online per-pixel traversability prediction during robot deployment with minimal human labeling. It combines DINOv2+FeatUp visual features, a novelty detector using Faiss, and sparse intra- and cross-image annotations to rapidly retrain a traversability head, addressing domain shift without teleoperation. The method is validated in a high-fidelity Unreal Engine simulator and on real-world data, achieving state-of-the-art performance with significantly less labeling overhead and demonstrating strong sim2real transfer. The approach yields practical benefits for field robotics by providing safe, context-aware navigation in unstructured settings and enabling rapid adaptation to new environments.

Abstract

The increasing use of robots in unstructured environments necessitates the development of effective perception and navigation strategies to enable field robots to successfully perform their tasks. In particular, it is key for such robots to understand where in their environment they can and cannot travel -- a task known as traversability estimation. However, existing geometric approaches to traversability estimation may fail to capture nuanced representations of traversability, whereas vision-based approaches typically either involve manually annotating a large number of images or require robot experience. In addition, existing methods can struggle to address domain shifts as they typically do not learn during deployment. To this end, we propose a human-in-the-loop (HiL) method for traversability estimation that prompts a human for annotations as-needed. Our method uses a foundation model to enable rapid learning on new annotations and to provide accurate predictions even when trained on a small number of quickly-provided HiL annotations. We extensively validate our method in simulation and on real-world data, and demonstrate that it can provide state-of-the-art traversability prediction performance.

Figures (4)

• Figure 1: Our proposed human-in-the-loop (HiL) traversability prediction method. A neural network built upon DINOv2 oquab2023dinov2 and FeatUp fu2024featup predicts the traversability of each pixel in images collected by a robot. A novelty detector flags novel images and requests sparse annotations from a human when an unfamiliar image is detected. The sparse, easily-provided HiL annotations are used for online learning during robot deployment.
• Figure 2: Overview of our proposed human-in-the-loop traversability prediction method. RGB camera images are passed through a DINOv2 oquab2023dinov2 feature extractor and FeatUp fu2024featup feature upsampler. The traversability prediction head takes upsampled image features as input and produces a traversability prediction (between 0 and 1) and an uncertainty score for each pixel. The class token from DINOv2 is fed to a novelty detector that detects novel images and requests labels for novel images from the HiL. The traversability prediction head is rapidly retrained when new HiL annotations are provided.
• Figure 3: Images from our simulator, showing the forest (top left), warehouse (top right), lot (bottom left), and dark lot (bottom right) environments.
• Figure 4: Traversability predictions using Big CHUNGUS, showing color images (top), predictions from a model trained only on data from our simulator (middle), and predictions from a model trained only on real-world data (bottom). The traversability values range from 0 (least traversable) to 1 (most traversable). Higher traversability is indicated by warmer colors.