Auditorily Embodied Conversational Agents: Effects of Spatialization and Situated Audio Cues on Presence and Social Perception

TL;DR

Auditory embodiment investigates whether audio-only cues—specifically spatialized voice and Foley sounds—can convey a conversational agent's bodily presence in users' spaces. A four-condition within-subject study with 24 participants demonstrates that spatialization and Foley increase co-presence, but Foley can impair attention, message understanding, and civil impressions, with some interaction effects. The findings reveal both the potential and limits of audio-based embodiment, underscoring the need to align embodied cues with social norms and perceptual constraints. These insights inform the design of auditory agents for social companionship, situated storytelling, and spatial navigation, while outlining deployment considerations for portable, in-the-wild systems.

Abstract

Embodiment can enhance conversational agents, such as increasing their perceived presence. This is typically achieved through visual representations of a virtual body; however, visual modalities are not always available, such as when users interact with agents using headphones or display-less glasses. In this work, we explore auditory embodiment. By introducing auditory cues of bodily presence - through spatially localized voice and situated Foley audio from environmental interactions - we investigate how audio alone can convey embodiment and influence perceptions of a conversational agent. We conducted a 2 (spatialization: monaural vs. spatialized) x 2 (Foley: none vs. Foley) within-subjects study, where participants (n=24) engaged in conversations with agents. Our results show that spatialization and Foley increase co-presence, but reduce users' perceptions of the agent's attention and other social attributes.

Figures (8)

• Figure 1: Agent activity sequences. (Top) shows the activities and their corresponding locations within the room, with the participant positioned at its center. (Bottom) shows the temporal ordering of the three activity sequences the agent followed in our experiment.
• Figure 2: Experimental conditions. The agent’s audio was rendered either monaurally(left) or spatialized(right), with the addition of Foley also varied: none(top) vs. Foley(bottom) (e.g., sounds of the agent pouring water). The visual depiction of the agent is for illustration only; participants experienced all conditions through audio alone.
• Figure 3: Study apparatus. We used ten OptiTrack motion-capture cameras (top left) to track a 4-marker rigid body attached to participants’ heads (top right). Participants engaged with the conversational agent using Shokz OpenRun Pro bone-conduction headphones (bottom left) and an Aisizon wireless lavalier microphone (bottom right).
• Figure 4: Effect of spatialization (top) and Foley (bottom) on social presence: co-presence (CP), attention allocation (AA), message understanding (MU), affective understanding (AU), and affective interdependence (AI).
• Figure 5: Effect of spatialization and Foley on likeability and social attraction.