Cross-Language Speaker Attribute Prediction Using MIL and RL

TL;DR

Cross-language speaker attribute prediction is challenged by linguistic variation, domain mismatch, and data imbalance. The authors extend the Reinforced Multiple Instance Learning (RL-MIL) framework with Domain Adversarial Training (DAT) to learn language-invariant utterance representations via a Gradient Reversal Layer, enabling cross-lingual transfer from high-resource to low-resource languages. Evaluations on a five-language Twitter corpus (few-shot) and a VoxCeleb2-derived 40-language zero-shot corpus show consistent Macro-F1 gains over MIL and the original RL-MIL, with gender prediction benefitting most (up to +0.17 Macro-F1, $p\le 0.01$) and age being more challenging. Zero-shot results indicate positive but less consistent improvements due to limited statistical power and greater unseen-language generalization difficulty. The study demonstrates that combining instance selection with adversarial domain adaptation is an effective strategy for robust cross-lingual speaker attribute prediction, with DAT identified as the primary driver of performance gains.

Abstract

We study multilingual speaker attribute prediction under linguistic variation, domain mismatch, and data imbalance across languages. We propose RLMIL-DAT, a multilingual extension of the reinforced multiple instance learning framework that combines reinforcement learning based instance selection with domain adversarial training to encourage language invariant utterance representations. We evaluate the approach on a five language Twitter corpus in a few shot setting and on a VoxCeleb2 derived corpus covering forty languages in a zero shot setting for gender and age prediction. Across a wide range of model configurations and multiple random seeds, RLMIL-DAT consistently improves Macro F1 compared to standard multiple instance learning and the original reinforced multiple instance learning framework. The largest gains are observed for gender prediction, while age prediction remains more challenging and shows smaller but positive improvements. Ablation experiments indicate that domain adversarial training is the primary contributor to the performance gains, enabling effective transfer from high resource English to lower resource languages by discouraging language specific cues in the shared encoder. In the zero shot setting on the smaller VoxCeleb2 subset, improvements are generally positive but less consistent, reflecting limited statistical power and the difficulty of generalizing to many unseen languages. Overall, the results demonstrate that combining instance selection with adversarial domain adaptation is an effective and robust strategy for cross lingual speaker attribute prediction.

Figures (10)

• Figure 1: Methodology workflow: extended RL-MIL framework with parallel DAT module for cross-lingual speaker attribute prediction.
• Figure 2: Test F1 by encoder and model, faceted by pooling (Twitter).
• Figure 3: Mean Test F1 Distribution by pooling (Twitter).
• Figure 4: Training F1 Score Learning Curves for MeanMLP on Gender Prediction (Twitter - Seed 42). This figure displays the training F1 learning curves from Weights & Biases for various model configurations predicting the gender attribute using the MeanMLP pooling head.
• Figure 5: Training F1 Score Learning Curves for MeanMLP on Age Prediction (Twitter - Seed 42). This figure displays the training F1 learning curves from Weights & Biases for various model configurations predicting the age attribute using the MeanMLP pooling head.