Dream-SLAM: Dreaming the Unseen for Active SLAM in Dynamic Environments

TL;DR

This work proposes a novel monocular active SLAM method, Dream-SLAM, which is based on dreaming cross-spatio-temporal images and semantically plausible structures of partially observed dynamic environments that outperforms state-of-the-art methods in localization accuracy, mapping quality, and exploration efficiency.

Abstract

In addition to the core tasks of simultaneous localization and mapping (SLAM), active SLAM additionally in- volves generating robot actions that enable effective and efficient exploration of unknown environments. However, existing active SLAM pipelines are limited by three main factors. First, they inherit the restrictions of the underlying SLAM modules that they may be using. Second, their motion planning strategies are typically shortsighted and lack long-term vision. Third, most approaches struggle to handle dynamic scenes. To address these limitations, we propose a novel monocular active SLAM method, Dream-SLAM, which is based on dreaming cross-spatio-temporal images and semantically plausible structures of partially observed dynamic environments. The generated cross-spatio-temporal im- ages are fused with real observations to mitigate noise and data incompleteness, leading to more accurate camera pose estimation and a more coherent 3D scene representation. Furthermore, we integrate dreamed and observed scene structures to enable long- horizon planning, producing farsighted trajectories that promote efficient and thorough exploration. Extensive experiments on both public and self-collected datasets demonstrate that Dream-SLAM outperforms state-of-the-art methods in localization accuracy, mapping quality, and exploration efficiency. Source code will be publicly available upon paper acceptance.

Figures (13)

• Figure 1: Dream-SLAM overview. Our pipeline consists of two main modules: localization and mapping, and exploration planning. (a) For localization, we propose to dream cross-spatio-temporal images, and use these images to construct additional 3D-2D foreground constraints that can effectively compensate for noise. For mapping, we propose a feedforward network to reconstruct per-pixel Gaussians of both static background and dynamic foreground. We further refine Gaussians based on multi-view constraints provided by cross-spatio-temporal and real images. (b) Our planning module dreams semantically plausible structures of unobserved areas. By integrating the dreamed and observed information, we plan a farsighted path, enabling an efficient and thorough exploration.
• Figure 2: Cross-spatio-temporal images for camera localization. (a) Traditional localization methods rely solely on the static background to estimate the camera pose. (b) In contrast, our method leverages both the dynamic foreground and static background by aligning the Gaussians' rendering at time $t$ with the dreamed cross-spatio-temporal image $I^t_{t+1}$, which represents the scene at time $t$ from the viewpoint of the $(t+1)$-th camera.
• Figure 3: Dreaming a cross-spatio-temporal image. Given the image $I_{t+1}$, we segment the foreground and dilate the foreground mask to obtain the inpainting mask $\mathbf{M}$. Then we feed the images $I_{t}$ and $I_{t+1}$, together with the mask $\mathbf{M}$, into the diffusion model, which dreams the cross-spatio-temporal image $I^t_{t+1}$.
• Figure 4: 3D Gaussian prediction and refinement. (a) Given images $I_{t+1}$ and $I_t$, we design a feedforward network to predict dynamic Gaussians at both time $t+1$ and time $t$. Here, we only visualize the predicted Gaussians at time $t+1$. (b) We refine Gaussians at time $t+1$ based on the photometric loss regarding both dreamed cross-spatio-temporal images $I_{t-1}^{t+1}, I_{t}^{t+1}$ and real image $I_{t+1}$. These images depict the same scene content at time $t+1$ from the $(t-1)$-th, $t$-th, and $(t+1)$-th views, respectively.
• Figure 5: Dreaming semantically plausible structures of unexplored areas. At an unvisited waypoint, we place virtual cameras to render images from different views and select the suitable images. Then we inpaint the selected images, and use them to predict Gaussians. By integrating the dreamed Gaussians into the existing Gaussians, we obtain more complete structures of the environment.