SpeechLess: Micro-utterance with Personalized Spatial Memory-aware Assistant in Everyday Augmented Reality

TL;DR

The paper tackles social friction and repetitive articulation in wearable AR assistants by introducing SpeechLess, a paradigm that uses personalized spatial memory to extrapolate missing intent from under-specified queries and enables gradations of speech granularity (Full/Partial/Zero). It implements a six-module pipeline (Query Decoding, Contextual Dimension Encoder, Memory Retriever, Response Composer, Personal Memory Store Manager, and User Verification) with memory-anchored retrieval and RAG-like response generation, validated through lab and in-the-wild studies. Results show reduced articulation effort and social discomfort with minimal degradation in perceived usability or accuracy, suggesting a practical path toward context-aware, privacy-conscious wearable AR assistance. This work lays groundwork for scalable, user-centric interaction models that balance contextual intelligence, privacy, and expressive control in everyday AR usage.

Abstract

Speaking aloud to a wearable AR assistant in public can be socially awkward, and re-articulating the same requests every day creates unnecessary effort. We present SpeechLess, a wearable AR assistant that introduces a speech-based intent granularity control paradigm grounded in personalized spatial memory. SpeechLess helps users "speak less," while still obtaining the information they need, and supports gradual explicitation of intent when more complex expression is required. SpeechLess binds prior interactions to multimodal personal context-space, time, activity, and referents-to form spatial memories, and leverages them to extrapolate missing intent dimensions from under-specified user queries. This enables users to dynamically adjust how explicitly they express their informational needs, from full-utterance to micro/zero-utterance interaction. We motivate our design through a week-long formative study using a commercial smart glasses platform, revealing discomfort with public voice use, frustration with repetitive speech, and hardware constraints. Building on these insights, we design SpeechLess, and evaluate it through controlled lab and in-the-wild studies. Our results indicate that regulated speech-based interaction, can improve everyday information access, reduce articulation effort, and support socially acceptable use without substantially degrading perceived usability or intent resolution accuracy across diverse everyday environments.

Figures (8)

• Figure 1: SpeechLess comprehends user intents using the benefits of both user-driven (explicit task prompting) and system-driven (proactive intent inference), intent resolution mechanisms.
• Figure 2: Conceptual illustration of device-to-user interface mapping. A minimalistic, hardware-based interface is preferred for the wearable; Human-in-the-loop for a transparent, trustworthy response.
• Figure 3: SpeechLess pipeline overview: Repeated queries (A) labeled as "commute routine" are stored in a personal memory data-storage (B), where a memory recall request (C) retrieves similar contextual memories, and uses them to infer the intention behind a user's new query, and proactively adapt to the user's current context (D). The context of bus schedule and personal routine are extrapolated from memories
• Figure 4: Varying granularity of speech inputs for intent expression.
• Figure 5: Proactive Intent Revision. SpeechLess can adapt a query using spatial memories. A query asking the sugar content of (A) can proactively be adapted for (B) upon a shift in the referent-of-interest.