REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting

TL;DR

REALM addresses the challenge of reasoning-based 3D segmentation under open-world conditions by grounding implicit language instructions in 3D Gaussian Splatting. The method combines a 3D feature field with an MLLM-based Visual Segmenter (LMSeg) and a Global-to-Local Spatial Grounding pipeline to produce accurate 3D masks from multi-view reasoning. The authors re-annotate LERF and 3D-OVS with implicit prompts and introduce REALM3D, showing state-of-the-art performance and practical 3D editing capabilities such as removal, replacement, and style transfer. This work narrows the gap between 2D reasoning in MLLMs and 3D spatial grounding, enabling robust open-world 3D understanding and interaction.

Abstract

Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility. Project page: https://ChangyueShi.github.io/REALM.

Figures (7)

• Figure 1: REALM vs. Direct Image Inputs. Feeding one or a few random rendered views into the MLLM makes the outcome highly sensitive to viewpoint selection (since our method relies on a voting strategy, it does not take effect when only 1 or 2 input views are provided).
• Figure 2: Overview of REALM.Top:Global-to-Local Spatial Grounding (GLSpaG) pipline hierarchically aggregates the outputs of LMSeg agents from global context to local refinement. Bottom left: We optimize a 3D feature field from 2D SAM masks for 3D consistent identification. Bottom right:MLLM-based Visual Segmenter (LMSeg) performs image-level reasoning on one viewpoint and integrates identity information from the optimized feature field to determine the selected instance ID.
• Figure 3: Global reasoning process. We visualize reasoning outputs of the MLLM for each global view.
• Figure 4: Qualitative Results on the LERF Dataset. The results demonstrate the ability of REALM to handle complex and implicit language queries with accurate visual grounding.
• Figure 5: Examples in REALM3D benchmark. We use MLLM bai2025qwen2 and SAM kirillov2023segment to annotate over 1K prompt–mask pairs, enabling quantitative evaluation on implicit queries.