Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems

TL;DR

The paper addresses the need for Spatial Intelligence in autonomous systems by proposing a unified, multi-modal pre-training framework that integrates camera, LiDAR, and additional sensors. It develops a taxonomy spanning single-modality, cross-modal, and unified pre-training, and connects foundation-model paradigms to open-world perception and planning through Generative World Models and Vision-Language-Action. Key contributions include a platform-centered analysis of datasets, a comprehensive survey of pre-training techniques, and a roadmap highlighting open challenges such as the semantic-geometric gap and deployment efficiency. The work demonstrates that unified multi-modal pre-training with open-world capabilities can significantly improve robustness, generalization, and planning for real-world autonomous systems, guiding future research toward scalable, trustworthy embodied AI.

Abstract

The rapid advancement of autonomous systems, including self-driving vehicles and drones, has intensified the need to forge true Spatial Intelligence from multi-modal onboard sensor data. While foundation models excel in single-modal contexts, integrating their capabilities across diverse sensors like cameras and LiDAR to create a unified understanding remains a formidable challenge. This paper presents a comprehensive framework for multi-modal pre-training, identifying the core set of techniques driving progress toward this goal. We dissect the interplay between foundational sensor characteristics and learning strategies, evaluating the role of platform-specific datasets in enabling these advancements. Our central contribution is the formulation of a unified taxonomy for pre-training paradigms: ranging from single-modality baselines to sophisticated unified frameworks that learn holistic representations for advanced tasks like 3D object detection and semantic occupancy prediction. Furthermore, we investigate the integration of textual inputs and occupancy representations to facilitate open-world perception and planning. Finally, we identify critical bottlenecks, such as computational efficiency and model scalability, and propose a roadmap toward general-purpose multi-modal foundation models capable of achieving robust Spatial Intelligence for real-world deployment.

Figures (7)

• Figure 1: Overview of the paper structure. We systematically structure the landscape of multi-modal data pre-training for forging Spatial Intelligence. This work is organized into four key pillars: (1) Background, introducing onboard sensors and foundational learning paradigms; (2) Platforms & Datasets, analyzing benchmarks across autonomous vehicles, drones, and other robotic systems; (3) Pre-Training Methodologies, categorized into single-modality, cross-modal (Camera/LiDAR-centric), and unified frameworks; and (4) Applications, highlighting downstream tasks from 3D perception to open-world planning.
• Figure 2: Chronological evolution of representative pre-training methods (2020--2025). The timeline illustrates the paradigm shift in representation learning for autonomous systems. Early approaches predominantly focused on single-modality self-supervision (e.g., LiDAR-only contrastive learning). In contrast, recent advancements (2023--present) demonstrate a surge in cross-modal synergy, characterized by Camera/LiDAR-centric methods and Unified pre-training frameworks, ultimately paving the way for generative world models and comprehensive spatial intelligence.
• Figure 3: Taxonomy of multi-modal pre-training methodologies. We structure the landscape into three pillars: (1) Platform-specific datasets, (2) Core pre-training techniques classified by sensor interaction (single-modality, cross-modal, and unified), and (3) Advanced open-world perception and planning tasks.
• Figure 4: Schematic illustration of representative LiDAR-only pre-training paradigms. To learn robust geometric representations from sparse point clouds without annotations, methods typically adopt three strategies: (a) Masked Autoencoding (MAE), which reconstructs missing structures to learn local geometry; (b) Contrastive Learning, which enforces view-invariant feature discrimination; and (c) Temporal Forecasting, which predicts future frames to capture dynamic scene evolution.
• Figure 5: Taxonomy of LiDAR-centric pre-training methodologies. To bridge the semantic gap of point clouds, these approaches leverage images as privileged information during training. The main paradigms involve: (a) Cross-modal MAE-based Pre-Training, which incorporates 2D-guided masking strategies to enhance geometric reconstruction and structural understanding; (b) Cross-modal Contrastive/Distillation Pre-Training, which either enforces feature alignment between modalities or directly transfers rich open-vocabulary semantics from pre-trained Vision Foundation Models (VFMs) to 3D encoders; and (c) Temporal Pre-Training, which exploits video-LiDAR sequences to capture motion dynamics and enforce spatiotemporal consistency.