Towards foundational LiDAR world models with efficient latent flow matching

TL;DR

This work tackles the transferability gap in LiDAR-based world models by proposing a foundational model trained on unlabeled outdoor LiDAR data and fine-tuned to diverse downstream tasks. It combines a high-ratio LiDAR compression VAE based on a Swin Transformer with a latent conditional flow matching (CFM) forecaster, achieving state-of-the-art performance with significantly reduced labeled data and improved computational efficiency. A representation-alignment strategy further enhances fine-tuning by aligning latent spaces across semantic and non-semantic domains. The results demonstrate strong cross-domain gains, reduced data requirements for semantic forecasting, and large speedups, highlighting practical implications for scalable, multi-domain LiDAR reasoning in autonomous systems.

Abstract

LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. Can we develop LiDAR world models that exhibit strong transferability across multiple domains? We conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse-beam & dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pre-trained model can achieve up to 11% absolute improvement (83% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability of dynamic learning significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceed the previous semantic occupancy forecasting models with only 5% of the labeled training data required by prior models. We also observed inefficiencies of current LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address this, we propose a latent conditional flow matching (CFM)-based frameworks that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model achieves SOTA performance on future-trajectory-conditioned semantic occupancy forecasting while being 23x more computationally efficient (a 28x FPS speedup); and achieves SOTA performance on semantic occupancy forecasting while being 2x more computationally efficient (a 1.1x FPS speedup).

Figures (13)

• Figure 1: Left: The overall pipeline of our method: we used the most readily publicly available LiDAR dataset from autonomous driving scenarios to train the proposed LiDAR world model. This well-trained world model is able to generalize well on the listed downstream tasks after fine-tuning, although scene and signal properties are quite different. Right: Comparison of our proposed world model with previous methods on nuScenes 4D semantic occupancy forecasting metrics of mIoU, IoU, and inference efficiency. Our approach achieves the best results in terms of both efficiency and performance.
• Figure 2: Architecture of our VAE for LiDAR compression. The model enables high compression ratios—exceeding those of previous methods—alongside high-fidelity reconstructions. Here we use $\bm{O}_v$ to represent the raw voxelized LiDAR points (Occupancy) and $\bm{O}_d$, $\bm{O}_s$ stand for densified or semantic labeled occupancy respectively.
• Figure 3: Architecture at the training stage of our proposed conditional velocity field predictor for time $t$. Historical frame latents are extracted via a frame‐wise VAE encoder, and noised future (target) latents are formed by injecting noise at timestep $t$. Latents are concatenated along the time dimension and passed through DiT blocks.
• Figure 4: IoU/mIoU comparison across 3s forecasting horizon of the presence of different fraction of fine-tuning data used from total data available between the various training procedures. In row order, each row refers to the results of (i) different beam adaptation (ii) outdoor-indoor adaptation, and (iii) semantic occupancy forecasting, respectively.
• Figure 5: The performance and efficiency of proposed VAE + CFM architecture under different NFE value during the sampling process. We observe that NFE=10 corresponds to the best model performance, with reasonable efficiency in terms of FPS and Gflops per frame.