CogniMap3D: Cognitive 3D Mapping and Rapid Retrieval

TL;DR

CogniMap3D tackles dynamic scene understanding by separating moving objects from static backgrounds while building a persistent memory of static environments. It combines a multi-stage motion cue framework, a cognitive mapping memory that stores and recalls static geometry and features, and a factor-graph based camera-trajectory optimization to refine poses and landmarks. The method demonstrates competitive depth estimation, robust camera pose estimation, and coherent 3D reconstruction across diverse datasets, with clear gains when revisiting previously mapped scenes due to memory recall and map fusion. This memory-augmented approach holds practical significance for autonomous navigation and augmented reality by enabling long-term, drift-resistant scene understanding in changing environments.

Abstract

We present CogniMap3D, a bioinspired framework for dynamic 3D scene understanding and reconstruction that emulates human cognitive processes. Our approach maintains a persistent memory bank of static scenes, enabling efficient spatial knowledge storage and rapid retrieval. CogniMap3D integrates three core capabilities: a multi-stage motion cue framework for identifying dynamic objects, a cognitive mapping system for storing, recalling, and updating static scenes across multiple visits, and a factor graph optimization strategy for refining camera poses. Given an image stream, our model identifies dynamic regions through motion cues with depth and camera pose priors, then matches static elements against its memory bank. When revisiting familiar locations, CogniMap3D retrieves stored scenes, relocates cameras, and updates memory with new observations. Evaluations on video depth estimation, camera pose reconstruction, and 3D mapping tasks demonstrate its state-of-the-art performance, while effectively supporting continuous scene understanding across extended sequences and multiple visits.

Figures (8)

• Figure 1: CogniMap3D maintains a cognitive mapping system that recalls, stores, and updates memories. Given an input video, it outputs camera poses and point clouds by isolating static scenes through motion cues, interacting with its memory bank, and optimizing across multiple visits.
• Figure 2: Multi-stage Motion Cue for Locating Dynamic Area. Given a pair of images in video, we first predict the initial depth and camera pose through VFM to establish 3D prior. Our pipeline then processes three specialized motion cues through progressive 2D-3D interaction effectively isolates robust dynamic regions, enabling accurate refinement and tracking across subsequent frames.
• Figure 3: Dynamic Mask Comparison. We visualize dynamic regions as white overlays on input images. Compared with MonST3R, our method achieves more complete and precise masks.
• Figure 4: Cognitive Mapping System. Given the input video, we estimate per‑frame dynamic mask with prior of the depth, confidence, camera pose. DINOv2 and Pointnet++ encode selected static images and static scene into latent features respectively. We then match features with a global feature table, if failed, a new memory slot is created; otherwise the corresponding memory is updated, enabling fast recall, relocalization, and refinement of the current scene.
• Figure 5: Qualitative Results of Dynamic 3D Reconstruction. We compare our method with concurrent works Monst3R monst3r and CUT3R cut3r. Our method achieves cleaner reconstructions with better preservation of both static and dynamic elements. The bottom rows demonstrate CogniMap3D's unique capability to store previous scenes in memory and recall them upon revisitation. We render stored memory scenes with higher brightness to distinguish them from newly observed scene.