From Speech-to-Spatial: Grounding Utterances on A Live Shared View with Augmented Reality

TL;DR

Speech-to-Spatial addresses the problem of grounding verbal remote guidance in AR without gesture or gaze by learning structured spatial patterns and building an object-centric relational graph. It uses an end-to-end pipeline in which spoken instructions are transcribed, objects are localized, a relational graph is constructed, and an AR indicator is anchored to the resolved referent, guided by LLM reasoning and memory augmentation. The approach yields measurable gains in task efficiency, cognitive load, and usability over voice-only baselines across remote maintenance, navigation, and personal-assistance contexts, while maintaining a lightweight, modality-lean grounding workflow. By converting disembodied speech into spatially explainable, actionable AR guidance, Speech-to-Spatial advances remote assistance toward persistent, interpretable Grounding with memory, enabling robust, scalable grounding without reliance on additional cues or manual annotations.

Abstract

We introduce Speech-to-Spatial, a referent disambiguation framework that converts verbal remote-assistance instructions into spatially grounded AR guidance. Unlike prior systems that rely on additional cues (e.g., gesture, gaze) or manual expert annotations, Speech-to-Spatial infers the intended target solely from spoken references (speech input). Motivated by our formative study of speech referencing patterns, we characterize recurring ways people specify targets (Direct Attribute, Relational, Remembrance, and Chained) and ground them to our object-centric relational graph. Given an utterance, referent cues are parsed and rendered as persistent in-situ AR visual guidance, reducing iterative micro-guidance ("a bit more to the right", "now, stop.") during remote guidance. We demonstrate the use cases of our system with remote guided assistance and intent disambiguation scenarios. Our evaluation shows that Speechto-Spatial improves task efficiency, reduces cognitive load, and enhances usability compared to a conventional voice-only baseline, transforming disembodied verbal instruction into visually explainable, actionable guidance on a live shared view.

Figures (10)

• Figure 1: End-to-end pipeline of Speech-to-Spatial: From speech with visual inputs and prior memories (if present), Speech-to-Spatial extracts the visible, linguistic attributes, associates them with a relational graph representation, and generates an AR visual indicator.
• Figure 2: Attribute parsing: Transcribed text of verbal instructions is extracted into a structure via LLM.
• Figure 3: Object-centric relational graph: Each object maintains a graph representation that holds Intra-object attributes--Space, Action, Intent, Actor, Time, and Metadata--and is connected with other objects nodes--Inter-object attributes.
• Figure 4: Three use case scenarios of Speech-to-Spatial. (A) Remote Maintenance; (B) Indoor Navigation; and (C) Personal AI assistant.
• Figure 5: Comparison of median task completion time per referencing pattern : (A) Locate Task, (B) Move Task result; *** $p<.001$, ** $p<.01$, * $p<.05$; Audio, Full, Summary.