Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

TL;DR

This work tackles predicting turn-taking and backchannel in human-machine conversations using linguistic, acoustic, and visual signals. It introduces MM-F2F, a large, de-identified tri-modal dataset collected from in-the-wild two-person videos, with word-level annotations for turn-taking and backchannels, and an end-to-end framework capable of processing any combination of text, audio, and video inputs. The framework employs uni-modal encoders and a flexible, low-rank fusion module with modality selection and random modality dropout to achieve state-of-the-art performance, including substantial gains in backchannel prediction. The dataset and code are publicly released, enabling future research toward more natural and robust face-to-face human-computer interactions, with potential extensions to richer visual cues and personalization.

Abstract

This paper addresses the gap in predicting turn-taking and backchannel actions in human-machine conversations using multi-modal signals (linguistic, acoustic, and visual). To overcome the limitation of existing datasets, we propose an automatic data collection pipeline that allows us to collect and annotate over 210 hours of human conversation videos. From this, we construct a Multi-Modal Face-to-Face (MM-F2F) human conversation dataset, including over 1.5M words and corresponding turn-taking and backchannel annotations from approximately 20M frames. Additionally, we present an end-to-end framework that predicts the probability of turn-taking and backchannel actions from multi-modal signals. The proposed model emphasizes the interrelation between modalities and supports any combination of text, audio, and video inputs, making it adaptable to a variety of realistic scenarios. Our experiments show that our approach achieves state-of-the-art performance on turn-taking and backchannel prediction tasks, achieving a 10% increase in F1-score on turn-taking and a 33% increase on backchannel prediction. Our dataset and code are publicly available online to ease of subsequent research.

Figures (7)

• Figure 1: In face-to-face conversation scenarios, the computer determines keeping, turn-taking and backchannel actions according to the user's linguistic, acoustic and visual signals (as shown above the line).
• Figure 2: Samples of EgoCom vs. our MM-F2F dataset and visualized distribution of our dataset. (a) Samples of EgoCom dataset. (b) Samples of our MM-F2F dataset.
• Figure 3: Architecture of our proposed multi-modal turn-taking and backchannel prediction framework. The text, audio and video inputs are first fed into uni-modal encoders $E_{T},E_{A},E_{V}$ to extract the uni-modal features $\boldsymbol{z_{T}},\boldsymbol{z_{A}},\boldsymbol{z_{V}}$ respectively. The features are then fused in the flexible fusion model. The fused feature $\boldsymbol{h}$ will be input into a prediction head to predict the output probability $\hat{y}$. Our framework supports uni-modal, bi-modal and tri-modal inputs of text, audio and video signals.
• Figure 4: Qualitative results on visual modal ablation. Dynamics of visual signals transmit communication cues that text and audio signals cannot capture.
• Figure 5: Failure case. When the speaker pauses to think while the talking context is semantically incomplete, our framework might sometimes mistakenly initiate turn-taking. In this case, in contrast, providing a backchannel would be more appropriate.