Multimodal Belief Prediction
John Murzaku, Adil Soubki, Owen Rambow
TL;DR
This paper pioneers multimodal belief prediction by integrating text and audio to infer speaker commitment, using the CB-Prosody corpus with audio start/end times and continuous belief annotations. It establishes strong baselines on acoustic-prosodic features, fine-tunes BERT for text and Whisper for audio, and introduces multimodal fusion architectures (early and late), with late fusion of BERT and Whisper achieving the best performance ($MAE$ $=0.62$, $\rho=0.83$). The key finding is that incorporating audio signals yields significant gains over text-only models, establishing a new state-of-the-art on CBP and highlighting the value of multimodal cues for event factuality belief prediction. The work lays groundwork for applying multimodal belief models to broader corpora and exploring multi-task setups with deception, potentially improving robustness and generalization in belief detection tasks.
Abstract
Recognizing a speaker's level of commitment to a belief is a difficult task; humans do not only interpret the meaning of the words in context, but also understand cues from intonation and other aspects of the audio signal. Many papers and corpora in the NLP community have approached the belief prediction task using text-only approaches. We are the first to frame and present results on the multimodal belief prediction task. We use the CB-Prosody corpus (CBP), containing aligned text and audio with speaker belief annotations. We first report baselines and significant features using acoustic-prosodic features and traditional machine learning methods. We then present text and audio baselines for the CBP corpus fine-tuning on BERT and Whisper respectively. Finally, we present our multimodal architecture which fine-tunes on BERT and Whisper and uses multiple fusion methods, improving on both modalities alone.