M3PT: A Transformer for Multimodal, Multi-Party Social Signal Prediction with Person-aware Blockwise Attention

TL;DR

This work tackles multi-party social signal prediction by introducing M3PT, a causal transformer with modality- and person-aware blockwise attention that processes multiple social cues across participants and time. Signals are tokenized per modality via modality-specific VQ-VAE codebooks, enabling a unified, discrete representation for a multi-person transformer to learn cross-person interactions. Evaluated on the HHCD triadic dining dataset, M3PT demonstrates that multimodal inputs improve bite timing and speaking status predictions, with ablations showing the importance of each modality and the impact of temporal context. The study advances socially aware robotics by providing a single, multimodal model capable of predicting discrete social signals in multi-party settings, while acknowledging limitations in predicting continuous signals and highlighting ethical considerations for deployment.

Abstract

Understanding social signals in multi-party conversations is important for human-robot interaction and artificial social intelligence. Social signals include body pose, head pose, speech, and context-specific activities like acquiring and taking bites of food when dining. Past work in multi-party interaction tends to build task-specific models for predicting social signals. In this work, we address the challenge of predicting multimodal social signals in multi-party settings in a single model. We introduce M3PT, a causal transformer architecture with modality and temporal blockwise attention masking to simultaneously process multiple social cues across multiple participants and their temporal interactions. We train and evaluate M3PT on the Human-Human Commensality Dataset (HHCD), and demonstrate that using multiple modalities improves bite timing and speaking status prediction. Source code: https://github.com/AbrarAnwar/masked-social-signals/.

Figures (2)

• Figure 1: Model Architecture. M3PT consists of a multi-party, multimodal causal transformer that attends to social signal features across people and timesteps. This transformer allows our model to reason about the interactants in multi-party settings over time. We encode these features with a VQ-VAE to tokenize continuous social signals for the transformer. We apply time-specific, person-specific, and modality-specific positional encodings along with a blockwise attenttion masking strategy, as shown in Fig. \ref{['fig:blockatten']} to allow for the transformer to learn the relationship between these inputs. Then, we reconstruct the discrete social signals, speaking status and bite timing.
• Figure 2: Blockwise Attention Masking: Conventional lower-triangular masking (left) would not be aware of the timestep and person-specific feature chunks present in multi-party, multimodal settings. Our blockwise attention mask (right) masks each person's social signals when they must be predicted, allowing the model to focus solely on capturing interactions from others' social signals.