Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Perception to Action: Spatial AI Agents and World Models

Gloria Felicia, Nolan Bryant, Handi Putra, Ayaan Gazali, Eliel Lobo, Esteban Rojas

TL;DR

This work proposes a unified three-axis taxonomy that connects agentic AI capabilities (Memory, Planning, Tool Use) with spatial intelligence tasks (Navigation, Scene Understanding, Manipulation, Geospatial Analysis) across micro, meso, and macro scales, addressing a fragmentation in existing surveys. By synthesizing insights from over 2,000 papers (742 cited), it highlights three design patterns—GNN-LLM integration, world-model planning, and memory-grounded perception—and introduces SpatialAgentBench as a framework for cross-domain evaluation. The authors articulate six grand challenges (unified spatial representation, grounded long-horizon planning, safe deployment, sim-to-real transfer, scalable multi-agent coordination, and efficient edge deployment) and outline six industry design patterns to guide future research and deployment in robotics, autonomous vehicles, and geospatial intelligence. The work aims to accelerate the development of spatially-aware autonomous systems by providing a common vocabulary, benchmark considerations, and a practical roadmap for integrating perception, memory, planning, and action across scales.

Abstract

While large language models have become the prevailing approach for agentic reasoning and planning, their success in symbolic domains does not readily translate to the physical world. Spatial intelligence, the ability to perceive 3D structure, reason about object relationships, and act under physical constraints, is an orthogonal capability that proves important for embodied agents. Existing surveys address either agentic architectures or spatial domains in isolation. None provide a unified framework connecting these complementary capabilities. This paper bridges that gap. Through a thorough review of over 2,000 papers, citing 742 works from top-tier venues, we introduce a unified three-axis taxonomy connecting agentic capabilities with spatial tasks across scales. Crucially, we distinguish spatial grounding (metric understanding of geometry and physics) from symbolic grounding (associating images with text), arguing that perception alone does not confer agency. Our analysis reveals three key findings mapped to these axes: (1) hierarchical memory systems (Capability axis) are important for long-horizon spatial tasks. (2) GNN-LLM integration (Task axis) is a promising approach for structured spatial reasoning. (3) World models (Scale axis) are essential for safe deployment across micro-to-macro spatial scales. We conclude by identifying six grand challenges and outlining directions for future research, including the need for unified evaluation frameworks to standardize cross-domain assessment. This taxonomy provides a foundation for unifying fragmented research efforts and enabling the next generation of spatially-aware autonomous systems in robotics, autonomous vehicles, and geospatial intelligence.

From Perception to Action: Spatial AI Agents and World Models

TL;DR

This work proposes a unified three-axis taxonomy that connects agentic AI capabilities (Memory, Planning, Tool Use) with spatial intelligence tasks (Navigation, Scene Understanding, Manipulation, Geospatial Analysis) across micro, meso, and macro scales, addressing a fragmentation in existing surveys. By synthesizing insights from over 2,000 papers (742 cited), it highlights three design patterns—GNN-LLM integration, world-model planning, and memory-grounded perception—and introduces SpatialAgentBench as a framework for cross-domain evaluation. The authors articulate six grand challenges (unified spatial representation, grounded long-horizon planning, safe deployment, sim-to-real transfer, scalable multi-agent coordination, and efficient edge deployment) and outline six industry design patterns to guide future research and deployment in robotics, autonomous vehicles, and geospatial intelligence. The work aims to accelerate the development of spatially-aware autonomous systems by providing a common vocabulary, benchmark considerations, and a practical roadmap for integrating perception, memory, planning, and action across scales.

Abstract

While large language models have become the prevailing approach for agentic reasoning and planning, their success in symbolic domains does not readily translate to the physical world. Spatial intelligence, the ability to perceive 3D structure, reason about object relationships, and act under physical constraints, is an orthogonal capability that proves important for embodied agents. Existing surveys address either agentic architectures or spatial domains in isolation. None provide a unified framework connecting these complementary capabilities. This paper bridges that gap. Through a thorough review of over 2,000 papers, citing 742 works from top-tier venues, we introduce a unified three-axis taxonomy connecting agentic capabilities with spatial tasks across scales. Crucially, we distinguish spatial grounding (metric understanding of geometry and physics) from symbolic grounding (associating images with text), arguing that perception alone does not confer agency. Our analysis reveals three key findings mapped to these axes: (1) hierarchical memory systems (Capability axis) are important for long-horizon spatial tasks. (2) GNN-LLM integration (Task axis) is a promising approach for structured spatial reasoning. (3) World models (Scale axis) are essential for safe deployment across micro-to-macro spatial scales. We conclude by identifying six grand challenges and outlining directions for future research, including the need for unified evaluation frameworks to standardize cross-domain assessment. This taxonomy provides a foundation for unifying fragmented research efforts and enabling the next generation of spatially-aware autonomous systems in robotics, autonomous vehicles, and geospatial intelligence.
Paper Structure (38 sections, 6 equations, 1 figure, 3 tables)

This paper contains 38 sections, 6 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Related Work and Survey Comparison
  4. Unified Three-Axis Taxonomy
  5. Taxonomy Axes
  6. Methods-Taxonomy Mapping
  7. Agentic AI Components for Spatial Intelligence
  8. Memory Systems: How Do Agents Remember Spatial Information?
  9. Planning Systems: How Do Agents Plan Over Spatial Horizons?
  10. Tool Use and Action: How Do Agents Ground Language in Geometry?
  11. Spatial Intelligence Domains
  12. Navigation: How Do Agents Move Through Space?
  13. Scene Understanding: How Do Agents Perceive 3D Structure?
  14. Manipulation: How Do Agents Interact with Objects?
  15. Geospatial Analysis: How Do Agents Reason at Planetary Scale?
  16. ...and 23 more sections

Figures (1)

  • Figure 1: A unified three-axis taxonomy connecting Agentic AI capabilities with Spatial Intelligence domains across spatial scales. The intersection of these dimensions defines the design space for autonomous spatial intelligence systems. Design trade-off: No existing method achieves strong performance across all three axes simultaneously. Micro-scale manipulation systems (bottom-left) achieve centimeter precision but cannot plan beyond the immediate workspace. Geospatial models (top-right) reason at planetary scale but lack closed-loop action capabilities. The sparsely populated regions of this space (e.g., macro-scale manipulation, micro-scale geospatial) represent open research opportunities where cross-axis integration could yield high-impact advances.