M3: 3D-Spatial MultiModal Memory

TL;DR

M3 introduces a spatial multimodal memory that fuses 3D Gaussian splatting with multiple foundation models to preserve and render multimodal knowledge for static scenes. It compresses high‑dimensional, cross‑model features into a compact Principal Scene Components memory and uses learnable Principle Queries with Gaussian Memory Attention to render model‑aligned embeddings in 3D space. The approach demonstrates improved memorization and downstream perception performance across diverse datasets and foundation models, while maintaining lower compute, and it is validated in real‑world indoor robot tasks. This work advances practical 3D feature distillation by enabling scalable, semantically rich, and interactable memory suitable for long‑horizon tasks and real‑time querying.

Abstract

We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-language models (VLMs), perception models, and large multimodal and language models (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3's feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

Figures (8)

• Figure 1: Our proposed MultiModal Memory integrates Gaussian splatting with foundation models to efficiently store multimodal memory in a Gaussian structure. The feature maps rendered by our approach exhibit high fidelity, preserving the strong expressive capabilities of the foundation models.
• Figure 2: A scene ($\mathbf{V}$) is composed of both structure ($\mathbf{S}$) and knowledge ($\mathbf{I}$). To model these, we leverage multiple foundation models to extract multi-granularity scene knowledge, and employ 3D Gaussian splatting to represent the spatial structure. By combining these techniques, we construct a spatial multimodal memory (M3), which enables downstream applications such as retrieval, captioning and grounding.
• Figure 3: Given a video sequence, we utilize foundation models ($\mathbf{F}$) to extract raw features ($\mathbf{R}$). These features are reduced using Algorithm \ref{['alg:sim']}, producing principal scene components ($\mathbf{PSC}$), which are stored in a memory bank. We introduce optimizable attribute queries ($q$) to Gaussian primitives, and apply a Gaussian Memory Attention ($\mathbf{A}_{gm}$) mechanism to produce the final rendered features ($\hat{\mathbf{R}}$), which can be linked back to various heads of the foundation models.
• Figure 4: The UMAP visualization of model embedding manifolds reveals distinct shapes, reflecting different focus.
• Figure 5: Illustration of patch-level visual embedding extraction their applications.