Multilingual Extraction and Recognition of Implicit Discourse Relations in Speech and Text
Ahmed Ruby, Christian Hardmeier, Sara Stymne
TL;DR
This work tackles implicit discourse relation recognition across text and speech in multilingual settings by constructing a multilingual multimodal dataset (English, French, Spanish) using an adapted MM-IDR pipeline. It introduces a Qwen2-Audio-based model that jointly encodes text and aligned audio, augmented with prosody and audio pooling statistics, and demonstrates that while text remains the strongest signal, multimodal fusion can improve performance for low-resource languages and cross-lingual transfer. The study provides rigorous dataset construction details, baseline comparisons, and extensive ablations showing language-specific fusion dynamics, and it suggests future opportunities to incorporate additional modalities and fusion control to further enhance IDR classification. Overall, the work advances multilingual multimodal discourse analysis and offers practical resources for cross-language discourse understanding and translation evaluation.
Abstract
Implicit discourse relation classification is a challenging task, as it requires inferring meaning from context. While contextual cues can be distributed across modalities and vary across languages, they are not always captured by text alone. To address this, we introduce an automatic method for distantly related and unrelated language pairs to construct a multilingual and multimodal dataset for implicit discourse relations in English, French, and Spanish. For classification, we propose a multimodal approach that integrates textual and acoustic information through Qwen2-Audio, allowing joint modeling of text and audio for implicit discourse relation classification across languages. We find that while text-based models outperform audio-based models, integrating both modalities can enhance performance, and cross-lingual transfer can provide substantial improvements for low-resource languages.