SpatialMem: Unified 3D Memory with Metric Anchoring and Fast Retrieval

TL;DR

SpatialMem tackles the challenge of building a persistent, metric-grounded 3D memory from monocular RGB video to support language-guided navigation and object retrieval in indoor environments. It introduces a hierarchical memory organized as a rooted tree with Level-1 anchors (walls, doors, windows), Level-2 objects, and Level-3 two-layer descriptions (attributes and anchor-relations), all anchored to a common $z$-aligned metric frame. The system emphasizes metric grounding, open-vocabulary semantics, and low-latency querying, enabling interpretable spatial reasoning over relations like distance and visibility. Experimental results across three real indoor scenes show strong layout understanding, competitive navigation, and robust object localization under clutter, with efficient memory updates suitable for real-time applications on commodity hardware. SpatialMem thus provides a practical, extensible framework for embodied spatial intelligence that grounds open-vocabulary queries in a stable 3D world model.

Abstract

We present SpatialMem, a memory-centric system that unifies 3D geometry, semantics, and language into a single, queryable representation. Starting from casually captured egocentric RGB video, SpatialMem reconstructs metrically scaled indoor environments, detects structural 3D anchors (walls, doors, windows) as the first-layer scaffold, and populates a hierarchical memory with open-vocabulary object nodes -- linking evidence patches, visual embeddings, and two-layer textual descriptions to 3D coordinates -- for compact storage and fast retrieval. This design enables interpretable reasoning over spatial relations (e.g., distance, direction, visibility) and supports downstream tasks such as language-guided navigation and object retrieval without specialized sensors. Experiments across three real-life indoor scenes demonstrate that SpatialMem maintains strong anchor-description-level navigation completion and hierarchical retrieval accuracy under increasing clutter and occlusion, offering an efficient and extensible framework for embodied spatial intelligence.

Figures (5)

• Figure 1: Performance and efficiency comparison of SpatialMem with multimodal baselines. Left: radar plot summarizing accuracy across relative-position reasoning and navigation, averaged over three scenes. Right: testing time vs success rate for navigation (top) and object retrieval (bottom), highlighting task latency and its trade-off with accuracy.
• Figure 2: Overview of our pipeline. Our pipeline has five steps: image ingestion normalizes egocentric RGB while preserving parallax and temporal cues; geometry estimation recovers intrinsics/extrinsics and dense depth (e.g., transformer-based) with light bundle adjustment; metric alignment detects the floor, aligns to global $z$, and sets scale via a height prior; structure and objects are detected, lifted to 3D, and associated to nearby anchors; finally, a rooted memory tree supports path-based relational queries over anchors and object nodes.
• Figure 3: Geometry back-end and metric alignment. We use a feed-forward RGB geometry module and align to an upright, metric frame.
• Figure 4: Tree-structured scene memory: anchors (Level-1), objects (Level-2), and two-layer text (Level-3).
• Figure 5: Overview of SpatialMem pipeline. The memory tree (center) organizes structural anchors, objects, and contextual relations. Left: navigation uses the anchor graph to resolve user/goal locations and generate step-by-step graph guidance. Right: object retrieval parses a language query, searches the tree, and returns an answer with linked 3D anchors and 2D evidence.