Analyzing Language Bias Between French and English in Conventional Multilingual Sentiment Analysis Models

TL;DR

The paper investigates language bias between English and French in conventional multilingual sentiment analysis using three balanced datasets across music, DVDs, and books. It employs SVM and Naive Bayes with TF-IDF features and assesses fairness with Fairlearn, reporting French in general achieving higher accuracy while Fairlearn metrics reveal mixed equity depending on the model and domain; SVM shows near-equitable Demographic Parity Ratios ($ ext{DPR}\approx 0.963$–$0.989$) but substantial Equalized Odds disparities in some domains, whereas Naive Bayes exhibits larger cross-language biases (DPR as low as $0.813$ and EOR as low as $0.352$). The study highlights the importance of equitable multilingual NLP as datasets expand to include more languages and domains, and it demonstrates how bias metrics can guide improvements beyond raw accuracy. The findings advocate for balanced data, domain-aware preprocessing, and careful fairness evaluation to ensure fair sentiment analysis across languages in real-world applications, especially as multilingual NLP scales. $\$

Abstract

Inspired by the 'Bias Considerations in Bilingual Natural Language Processing' report by Statistics Canada, this study delves into potential biases in multilingual sentiment analysis between English and French. Given a 50-50 dataset of French and English, we aim to determine if there exists a language bias and explore how the incorporation of more diverse datasets in the future might affect the equity of multilingual Natural Language Processing (NLP) systems. By employing Support Vector Machine (SVM) and Naive Bayes models on three balanced datasets, we reveal potential biases in multilingual sentiment classification. Utilizing Fairlearn, a tool for assessing bias in machine learning models, our findings indicate nuanced outcomes. With French data outperforming English across accuracy, recall, and F1 score metrics in both models, hinting at a language bias favoring French. However, Fairlearn's metrics suggest that the SVM approaches equitable levels with a demographic parity ratio of 0.963, 0.989, and 0.985 for the three separate datasets, indicating near-equitable treatment across languages. In contrast, Naive Bayes demonstrates greater disparities, evidenced by a demographic parity ratio of 0.813, 0.908, and 0.961. These findings reveal the importance of developing equitable multilingual NLP systems, particularly as we anticipate the inclusion of more datasets in various languages in the future.