Collaborative Learning of Local 3D Occupancy Prediction and Versatile Global Occupancy Mapping

TL;DR

This paper proposes Long-term Memory Prior Occupancy (LMPOcc), a plug-and-play framework that incorporates global occupancy priors to boost local prediction and simultaneously updates global maps with new observations, and introduces a model-agnostic prior format to enable continual updating of global occupancy and ensure compatibility across diverse prediction baselines.

Abstract

Vision-based 3D semantic occupancy prediction is vital for autonomous driving, enabling unified modeling of static infrastructure and dynamic agents. Global occupancy maps serve as long-term memory priors, providing valuable historical context that enhances local perception. This is particularly important in challenging scenarios such as occlusion or poor illumination, where current and nearby observations may be unreliable or incomplete. Priors aggregated from previous traversals under better conditions help fill gaps and enhance the robustness of local 3D occupancy prediction. In this paper, we propose Long-term Memory Prior Occupancy (LMPOcc), a plug-and-play framework that incorporates global occupancy priors to boost local prediction and simultaneously updates global maps with new observations. To realize the information gain from global priors, we design an efficient and lightweight Current-Prior Fusion module that adaptively integrates prior and current features. Meanwhile, we introduce a model-agnostic prior format to enable continual updating of global occupancy and ensure compatibility across diverse prediction baselines. LMPOcc achieves state-of-the-art local occupancy prediction performance validated on the Occ3D-nuScenes benchmark, especially on static semantic categories. Furthermore, we verify LMPOcc's capability to build large-scale global occupancy maps through multi-vehicle crowdsourcing, and utilize occupancy-derived dense depth to support the construction of 3D open-vocabulary maps. Our method opens up a new paradigm for continuous global information updating and storage, paving the way towards more comprehensive and scalable scene understanding in large outdoor environments.

Figures (7)

• Figure 1: Comparison of temporal information integration methods in 3D occupancy prediction. (a) Existing works primarily integrate information from adjacent observations. (b) Our work fuses perceptual information obtained from historical traversals of the current location. The historical perceptual information constructs global occupancy and serves as long-term memory priors.
• Figure 2: An overview of our LMPOcc framework. LMPOcc fristly generates Current Latent Features from surround-view images. Then it extracts spatially-aligned Prior Features from global occupancy and integrates them via the Current-Prior Fusion Module to generate Refined Latent Features. The refined latent features decode current occupancy logits, which are stored into corresponding locations in the global occupancy after visibility masking. Existing occupancy priors at these locations are replaced by the updated logits. Finally, the occupancy logits are converted into local current occupancy prediction results.
• Figure 3: Demonstration of using a 3D open vocabulary map to interact with a Vision-Language Model (VLM). A viewpoint along the navigation route is selected, from which a semantic map is rendered by the 3D open vocabulary map and then interpreted by the VLM. This enables the VLM to anticipate upcoming road conditions and make informed driving decisions to handle challenging scenarios. The example dialogue shows the VLM analyzing the scene and recommending appropriate actions.
• Figure 4: Visualization results of a region within our global occupancy. The left side shows the top view, and the right side shows the front view.
• Figure 5: Visualization results of global occupancy construction via crowdsourcing methodologies. Three collaborative agents construct the global occupancy map through crowdsourcing. (a) and (b) show the intermediate stages of the occupancy construction process. (c) displays the crowdsourced mapping result. Three colors mark the areas mapped by each agent.