Embodied Natural Language Interaction (NLI): Speech Input Patterns in Immersive Analytics

TL;DR

This work investigates how embodiment cues shape speech-based interaction in immersive analytics. By conducting a Wizard-of-Oz VR study with 15 participants, the authors quantify Speech Input Uncertainty via semantic entropy and classify utterances into five patterns based on Embodiment Reliance across Foraging, Sensemaking, and Action phases. The study introduces Embodied NLI terminology, an analysis framework, and practical design implications for speech-driven immersive tools, highlighting when users rely on embodied cues versus linguistic expressions. Findings suggest dynamic blending of embodied and non-embodied speech acts across tasks, informing adaptive interfaces and uncertainty-aware AI assistants in XR data visualization. The work lays groundwork for integrating intelligence and embodiment-aware speech in generative AI-powered immersive analytics tools.

Abstract

Embodiment shapes how users verbally express intent when interacting with data through speech interfaces in immersive analytics. Despite growing interest in Natural Language Interaction (NLI) for visual analytics in immersive environments, users' speech patterns and their use of embodiment cues in speech remain underexplored. Understanding their interplay is crucial to bridging the gap between users' intent and an immersive analytic system. To address this, we report the results from 15 participants in a user study conducted using the Wizard of Oz method. We performed axial coding on 1,280 speech acts derived from 734 utterances, examining how analysis tasks are carried out with embodiment and linguistic features. Next, we measured speech input uncertainty for each analysis task using the semantic entropy of utterances, estimating how uncertain users' speech inputs appear to an analytic system. Through these analyses, we identified five speech input patterns, showing that users dynamically blend embodied and non-embodied speech acts depending on data analysis tasks, phases, and embodiment reliance driven by the counts and types of embodiment cues in each utterance. We then examined how these patterns align with user reflections on factors that challenge speech interaction during the study. Finally, we propose design implications aligned with the five patterns.

Figures (10)

• Figure 1: Speech Input Analysis Framework.
• Figure 2: Speech Input Uncertainty per task, quantified as semantic entropy.
• Figure 3: The plots show the distribution of speech acts based on their semantic embedding distances from the centroid. 1) Y-axis: Semantic distance from the centroid; X-axis: Density of speech acts with the same distance, normalized between 0 and 1. 2) A broader, smoother, right-skewed distribution indicates increased variability and semantic entropy.
• Figure 4: Embodiment Cue: Count by Analysis Task and Phase. Each cell indicates the number of speech acts, categorized based on how many embodiment cues are presented within each speech act. Darker colors represent greater reliance on multiple, single, or non-embodiment cues when performing a task.
• Figure 5: Visualization of the five Speech Input Patterns, aligned with corresponding tasks, strategies, and embodiment usage. \ref{['fig:teaser']} presents the full context of task grouping and pattern distribution.