MCRL4OR: Multimodal Contrastive Representation Learning for Off-Road Environmental Perception

TL;DR

This work tackles off-road environmental perception by introducing MCRL4OR, a CLIP-like multimodal contrastive pre-training framework that jointly learns visual, action, and locomotion encoders. By aligning (observation+action) with locomotion, the model captures terrain-driven dynamics and yields transferable representations validated across cross-modal retrieval, dynamics prediction, and semantic segmentation on off-road data. The approach is demonstrated on the large-scale TartanDrive dataset and generalizes to ORFD for segmentation, with comprehensive ablations and reproducibility resources. Overall, MCRL4OR advances self-supervised multimodal learning for robust off-road perception and downstream robotics tasks.

Abstract

Most studies on environmental perception for autonomous vehicles (AVs) focus on urban traffic environments, where the objects/stuff to be perceived are mainly from man-made scenes and scalable datasets with dense annotations can be used to train supervised learning models. By contrast, it is hard to densely annotate a large-scale off-road driving dataset manually due to the inherently unstructured nature of off-road environments. In this paper, we propose a Multimodal Contrastive Representation Learning approach for Off-Road environmental perception, namely MCRL4OR. This approach aims to jointly learn three encoders for processing visual images, locomotion states, and control actions by aligning the locomotion states with the fused features of visual images and control actions within a contrastive learning framework. The causation behind this alignment strategy is that the inertial locomotion state is the result of taking a certain control action under the current landform/terrain condition perceived by visual sensors. In experiments, we pre-train the MCRL4OR with a large-scale off-road driving dataset and adopt the learned multimodal representations for various downstream perception tasks in off-road driving scenarios. The superior performance in downstream tasks demonstrates the advantages of the pre-trained multimodal representations. The codes can be found in \url{https://github.com/1uciusy/MCRL4OR}.

Figures (8)

• Figure 1: Overall framework of MCRL4OR, which jointly learn a visual observation encoder, a control action encoder, and a locomotion state encoder with the goal of predicting the correct correspondence relationships in a batch of multimodal training samples.
• Figure 2: Construction diagram of triplet samples, where a sample consists of a frame of visual image and its following 6s (240 frames) time series of locomotion states and control actions. Every adjacent samples have a 2 second stride.
• Figure 3: Some off-road image samples in our work. (a) the images from TartanDrive TartanDrive. (b) the images from ORFD ORFD. (c) the annotations of semantic segmentation in ORFD, white area for traversable, black area for non-traversable and gray for non-reachable.
• Figure 4: The left column is linear accelerations along three axes in locomotion state. The blue, orange and green lines are corresponded to linear acceleration along z, y and x axes. Right are top 4 similar observations retrieved. The paired positive image are bounded by a red frame.
• Figure 5: Left is linear acceleration along three axes in locomotion state. The blue, orange and green lines are corresponded to linear acceleration along z, y and x axes, the mean throttle is recorded along with the linear acceleration. Right are top 4 similar observations retrieved. The paired positive image are bounded by a red frame.