VertiFormer: A Data-Efficient Multi-Task Transformer for Off-Road Robot Mobility

TL;DR

VertiFormer tackles the problem of learning kinodynamic representations for off-road robot mobility under extreme data scarcity. It introduces a data-efficient, multi-task Transformer with a unified multi-modal latent representation, learnable masking, and non-autoregressive prediction to jointly model forward kinodynamics, inverse kinodynamics, and behavior cloning from only one hour of data. The approach is validated through extensive ablations and hardware experiments, showing that VertiFormer outperforms specialized baselines and end-to-end architectures while offering robust zero-shot capabilities when modalities are missing. The work demonstrates practical on-board applicability for navigating rugged vertical terrains and provides empirical guidance for training Transformers under limited robotics data regimes.

Abstract

Sophisticated learning architectures, e.g., Transformers, present a unique opportunity for robots to understand complex vehicle-terrain kinodynamic interactions for off-road mobility. While internet-scale data are available for Natural Language Processing (NLP) and Computer Vision (CV) tasks to train Transformers, real-world mobility data are difficult to acquire with physical robots navigating off-road terrain. Furthermore, training techniques specifically designed to process text and image data in NLP and CV may not apply to robot mobility. In this paper, we propose VertiFormer, a novel data-efficient multi-task Transformer model trained with only one hour of data to address such challenges of applying Transformer architectures for robot mobility on extremely rugged, vertically challenging, off-road terrain. Specifically, VertiFormer employs a new learnable masked modeling and next token prediction paradigm to predict the next pose, action, and terrain patch to enable a variety of off-road mobility tasks simultaneously, e.g., forward and inverse kinodynamics modeling. The non-autoregressive design mitigates computational bottlenecks and error propagation associated with autoregressive models. VertiFormer's unified modality representation also enhances learning of diverse temporal mappings and state representations, which, combined with multiple objective functions, further improves model generalization. Our experiments offer insights into effectively utilizing Transformers for off-road robot mobility with limited data and demonstrate our efficiently trained Transformer can facilitate multiple off-road mobility tasks onboard a physical mobile robot.

Figures (11)

• Figure 0: VertiFormer Architecture. VertiFormer employs a TransformerEncoder (left) to receive a history of terrain patches, actions, and poses along with multiple context tokens. To predict future states, the model computes cross-attention between these context tokens and the masked upcoming actions or poses. Causal masking is implemented during this cross-attention computation to ensure that predictions are conditioned only on past and present information, preventing information leakage from future time steps.
• Figure 1: Positional Encoding: Sinusoidal positional encoding achieves better model accuracy than learnable encoding for predicting $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{Z}$ components of the robot pose.
• Figure 2: Normalizing Output: Normalizing the Transformer output before passing the embeddings to the task decoder improves model performance.
• Figure 3: Kinodynamics Understanding: Without unified latent representation the model cannot capture temporal dependencies and understand kinodynamic transitions, resulting in an almost flat learning curve.
• Figure 4: Prediction Horizon:VertiFormer is capable of predicting a longer horizon without losing much accuracy due to its non-autoregressive nature.