The FIGNEWS Shared Task on News Media Narratives
Wajdi Zaghouani, Mustafa Jarrar, Nizar Habash, Houda Bouamor, Imed Zitouni, Mona Diab, Samhaa R. El-Beltagy, Muhammed AbuOdeh
TL;DR
The paper presents FIGNEWS, a multilingual shared task addressing bias and propaganda annotation in news posts about the Israel War on Gaza, leveraging a diverse corpus across English, French, Arabic, Hebrew, and Hindi. It details data collection from CrowdTangle, batch-wise annotation with two subtasks (bias and propaganda), and four evaluation tracks (Guidelines, IAA Quality, Quantity, Consistency), enabling cross-team collaboration and rigorous quality assessment. The study reports participation by 17 qualified teams and 85 bias annotators plus 51 propaganda annotators, producing 129,800 data points, with detailed results and observations on agreement, label distributions, and correlations. The work contributes robust guidelines, a comprehensive multilingual dataset, and insights for improving subjective annotation in NLP, with implications for media literacy, bias/detection research, and future multilingual expansion.
Abstract
We present an overview of the FIGNEWS shared task, organized as part of the ArabicNLP 2024 conference co-located with ACL 2024. The shared task addresses bias and propaganda annotation in multilingual news posts. We focus on the early days of the Israel War on Gaza as a case study. The task aims to foster collaboration in developing annotation guidelines for subjective tasks by creating frameworks for analyzing diverse narratives highlighting potential bias and propaganda. In a spirit of fostering and encouraging diversity, we address the problem from a multilingual perspective, namely within five languages: English, French, Arabic, Hebrew, and Hindi. A total of 17 teams participated in two annotation subtasks: bias (16 teams) and propaganda (6 teams). The teams competed in four evaluation tracks: guidelines development, annotation quality, annotation quantity, and consistency. Collectively, the teams produced 129,800 data points. Key findings and implications for the field are discussed.