Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR

Lixing Guo, Tobias Höllerer

TL;DR

Words into World presents a task-adaptive AR agent that marries open-vocabulary language understanding with depth-aware 3D grounding to support language-driven spatial retrieval in live environments. The system builds dynamic 3D scene graphs with rich relational predicates and uses a task-driven controller to select appropriate spatial and relational tools, enabling centimeter-scale localization, reliable relational reasoning, and responsive AR overlays on commodity headsets. GroundedAR-Bench provides a standardized evaluation suite for language-conditioned spatial grounding, 3D localization, and relational reasoning, with results showing significant gains over 2D baselines and robust performance across cluttered indoor scenes. The work argues for a practical blueprint where MLLMs contribute semantic breadth while coordinate-aware perception and explicit scene graphs deliver reliable spatial action, enabling actionable guidance in AR while addressing latency and safety concerns.

Abstract

Traditional augmented reality (AR) systems predominantly rely on fixed class detectors or fiducial markers, limiting their ability to interpret complex, open-vocabulary natural language queries. We present a modular AR agent system that integrates multimodal large language models (MLLMs) with grounded vision models to enable relational reasoning in space and language-conditioned spatial retrieval in physical environments. Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities, ranging from simple object identification to multi-object relational reasoning, while returning meter-accurate 3D anchors. It constructs dynamic AR scene graphs encoding nine typed relations (spatial, structural-semantic, causal-functional), enabling MLLMs to understand not just what objects exist, but how they relate and interact in 3D space. Through task-adaptive region-of-interest highlighting and contextual spatial retrieval, the system guides human attention to information-dense areas while supporting human-in-the-loop refinement. The agent dynamically invokes coordinate-aware tools for complex queries-selection, measurement, comparison, and actuation-grounding language understanding in physical operations. The modular architecture supports plug-and-use vision-language models without retraining, establishing AR agents as intermediaries that augment MLLMs with real-world spatial intelligence for interactive scene understanding. We also introduce GroundedAR-Bench, an evaluation framework for language-driven real world localization and relation grounding across diverse environments.

Words into World: A Task-Adaptive Agent for Language-Guided Spatial Retrieval in AR

TL;DR

Words into World presents a task-adaptive AR agent that marries open-vocabulary language understanding with depth-aware 3D grounding to support language-driven spatial retrieval in live environments. The system builds dynamic 3D scene graphs with rich relational predicates and uses a task-driven controller to select appropriate spatial and relational tools, enabling centimeter-scale localization, reliable relational reasoning, and responsive AR overlays on commodity headsets. GroundedAR-Bench provides a standardized evaluation suite for language-conditioned spatial grounding, 3D localization, and relational reasoning, with results showing significant gains over 2D baselines and robust performance across cluttered indoor scenes. The work argues for a practical blueprint where MLLMs contribute semantic breadth while coordinate-aware perception and explicit scene graphs deliver reliable spatial action, enabling actionable guidance in AR while addressing latency and safety concerns.

Abstract

Traditional augmented reality (AR) systems predominantly rely on fixed class detectors or fiducial markers, limiting their ability to interpret complex, open-vocabulary natural language queries. We present a modular AR agent system that integrates multimodal large language models (MLLMs) with grounded vision models to enable relational reasoning in space and language-conditioned spatial retrieval in physical environments. Our adaptive task agent coordinates MLLMs and coordinate-aware perception tools to address varying query complexities, ranging from simple object identification to multi-object relational reasoning, while returning meter-accurate 3D anchors. It constructs dynamic AR scene graphs encoding nine typed relations (spatial, structural-semantic, causal-functional), enabling MLLMs to understand not just what objects exist, but how they relate and interact in 3D space. Through task-adaptive region-of-interest highlighting and contextual spatial retrieval, the system guides human attention to information-dense areas while supporting human-in-the-loop refinement. The agent dynamically invokes coordinate-aware tools for complex queries-selection, measurement, comparison, and actuation-grounding language understanding in physical operations. The modular architecture supports plug-and-use vision-language models without retraining, establishing AR agents as intermediaries that augment MLLMs with real-world spatial intelligence for interactive scene understanding. We also introduce GroundedAR-Bench, an evaluation framework for language-driven real world localization and relation grounding across diverse environments.

Paper Structure

This paper contains 79 sections, 1 equation, 2 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision--Language Models for 3D Scene Understanding
  4. Open-Vocabulary Perception and Object Placement in AR
  5. Language-Guided AR Interaction and Agentic Systems
  6. Method
  7. Problem Formulation
  8. Layered Agent Architecture
  9. Sensing & Actuation Layer (on-device)
  10. Inputs
  11. Outputs
  12. Tool Layer
  13. Language input & planning tools
  14. Perception & 3D grounding tools
  15. Geo-anchoring & navigation tools
  16. ...and 64 more sections

Figures (2)

  • Figure 1: Overview of the Words into World architecture. Voice queries and RGB views from Meta Quest 3 flow through five layers: Sensing captures passthrough RGB, depth, and geolocation; Orchestrator interprets tasks and coordinates agents; Agent & Tool handles dialog, grounding, and navigation; World Model maintains scene graphs, relations, and geo-information; Feedback renders visual overlays, haptic cues, and audio guidance.
  • Figure 2: Qualitative example of Words into World on a desk scene. The system uses an open-vocabulary detector to identify objects (e.g., macbook laptop, curved monitor, LED desk lamp) and renders 3D wireframe bounding boxes aligned with the physical objects in the user's AR view.