Zero-Shot Adaptation to Robot Structural Damage via Natural Language-Informed Kinodynamics Modeling

TL;DR

This work proposes Zero-shot Language Informed Kinodynamics (ZLIK), which employs self-supervised learning to ground semantic information of damage descriptions in kinodynamic behaviors to learn a forward kinodynamics model in a data-driven manner.

Abstract

High-performance autonomous mobile robots endure significant mechanical stress during in-the-wild operations, e.g., driving at high speeds or over rugged terrain. Although these platforms are engineered to withstand such conditions, mechanical degradation is inevitable. Structural damage manifests as consistent and notable changes in kinodynamic behavior compared to a healthy vehicle. Given the heterogeneous nature of structural failures, quantifying various damages to inform kinodynamics is challenging. We posit that natural language can describe and thus capture this variety of damages. Therefore, we propose Zero-shot Language Informed Kinodynamics (ZLIK), which employs self-supervised learning to ground semantic information of damage descriptions in kinodynamic behaviors to learn a forward kinodynamics model in a data-driven manner. Using the high-fidelity soft-body physics simulator BeamNG.tech, we collect data from a variety of structurally compromised vehicles. Our learned model achieves zero-shot adaptation to different damages with up to 81% reduction in kinodynamics error and generalizes across the sim-to-real and full-to-1/10$^{\text{th}}$ scale gaps.

Figures (5)

• Figure 0: (a) Constructing a representation space for damage that is both semantically descriptive and kinodynamically grounded. (b) zlik leverages a Transformer Encoder-Decoder structure to approximate damaged kinodynamics.
• Figure 1: zlik's Transformer Encoder uses a two-stage attention architecture for both time (gray) and state dimension (orange).
• Figure 2: zlik outperforms Monolithic Transformer in terms of MSE and standard deviation across state dimensions for Fall and Multiple Tires Punctured & Suspensions Broken.
• Figure 3: Healthy Physical Robot Platform, V4W (middle), Used in the Cross-Embodiment Generalization Experiment, Compared with the Damaged V4W Variants, Front Left Tire Punctured (left) and Rear Left Wheel Removed (right).
• Figure 4: Cross-Embodiment Evaluation on a Physical 1/10$^{\text{th}}$ Scale V4W Robot. zlik achieves the best performance in zero-shot on two unseen damage classes in most dimensions.