Effect of Avatar Head Movement on Communication Behaviour, Experience of Presence and Conversation Success in Triadic Conversations

TL;DR

The study investigates ecologically valid evaluation of hearing devices by using triadic conversations in a virtual reality telepresence setting with avatars that vary in head movement realism. A 2×4 factorial design manipulated avatar animation (stat, auto, trans, video) and background noise, with 16 normal-hearing adults; outcomes included speech and motion metrics from microphones and motion capture, plus subjective presence and conversation-success ratings. Noise increased speech level by about 10.6 dB; higher avatar animation increased head orientation range and perceived realism, with video showing the strongest effects, while gaps/overlaps and overall conversation success were less affected by animation. The results imply that realistic interlocutor nonverbal cues are important for ecological validity in hearing-device evaluations, but additional cues such as facial expressions and gestures may be needed for full natural communication; limitations include a small, young, NH sample and triadic VR setup.

Abstract

Interactive communication in virtual reality can be used in experimental paradigms to increase the ecological validity of hearing device evaluations. This requires the virtual environment to elicit natural communication behaviour in listeners. This study evaluates the effect of virtual animated characters' head movements on participants' communication behaviour and experience. Triadic conversations were conducted between a test participant and two confederates. To facilitate the manipulation of head movements, the conversation was conducted in telepresence using a system that transmitted audio, head movement data and video with low delay. The confederates were represented by virtual animated characters (avatars) with different levels of animation: Static heads, automated head movement animations based on speech level onsets, and animated head movements based on the transmitted head movements of the interlocutors. A condition was also included in which the videos of the interlocutors' heads were embedded in the visual scene. The results show significant effects of animation level on the participants' speech and head movement behaviour as recorded by physical sensors, as well as on the subjective sense of presence and the success of the conversation. The largest effects were found for the range of head orientation during speech and the perceived realism of avatars. Participants reported that they were spoken to in a more helpful way when the avatars showed head movements transmitted from the interlocutors than when the avatars' heads were static. We therefore conclude that the representation of interlocutors must include sufficiently realistic head movements in order to elicit natural communication behaviour.

Figures (9)

• Figure 1: Top panel: top view of the virtual conversation setup. The test participant is virtually sitting on the empty chair. Centre panel: view of the virtual environment from the participant's perspective. Bottom panel: view of the virtual environment with real-time video textures in front of the avatars' faces, which was used in the condition 'video'.
• Figure 2: Scheme of possible combinations of speech segments in a triadic conversation. Each row represents one speaker, the dark grey bars are speech activity over time. Gaps and overlaps are marked by light blue bars. Only the times of speech gaps and overlaps were defined as turn takes.
• Figure 3: Median utterance duration during conversation of participants Subj and confederates Mar and Joa. Note that only participants, not the two confederates, were exposed to different visual conditions. Four outliers are outside the displayed x value range. Mar: in trans (n) $6.46\mathrm{\,s}$, in stat (n) $7.83\mathrm{\,s}$, Subj: in trans (n) $6.36\mathrm{\,s}$, in stat (n) $7.33\mathrm{\,s}$.
• Figure 4: Median duration of speech overlaps (left) and gaps (right) at turn takes between the three interlocutors. The speech overlap duration is noted in negative values. Note that the frequency of overlaps decreased in noise, whereas the frequency of gaps increased (Not shown here). Right panel: One outlier is outside the displayed x value range (in auto (n): 1.54 s).
• Figure 5: Relative occurrence over head yaw angle for each participant during listening (dark grey) and speaking (light grey), pooled over all conditions. The vertical lines indicate the position of the avatar Mar (M), the avatar Joa (J), and the centre between both. The avatars' positions are symmetric around 5 ° in lab coordinates. Occurrence values sum up to 1, and the angle is divided into increments of 2.5 °. Definition of listening and speaking: see section \ref{['sec:Analyses']}.