News Ninja: Gamified Annotation of Linguistic Bias in Online News

TL;DR

News Ninja tackles the scalability challenge of annotating linguistic bias in online news by combining education and data collection in a gamified framework. The system uses a tutorial, two data-annotation mechanics, five game modes, and direct/delayed feedback to train players and produce bias labels, achieving an inter-annotator agreement (IAA) on par with expert datasets ($$0.39$$) and surpassing crowdsourced baselines ($$0.21$$) with its $0.44$ score on BABE sentences. New sentences annotated by players reach IAA levels comparable to experts ($$0.399$$), and player labels achieve $79.8 ext{%}$ accuracy against new expert labels, indicating high data quality. The approach demonstrates scalable, education-enhancing crowdsourcing for linguistic bias datasets, with potential for continuous dataset updates and application to other NLP annotation tasks, while highlighting considerations around ground truth definitions and cultural biases.

Abstract

Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.

Figures (7)

• Figure 1: The figure shows how News Ninja applies the Gamified Knowledge Encoding Model to gamify bias learning and facilitate annotation through four steps. First, News Ninja breaks down learning objectives into knowledge units. Those units are presented to players through game mechanics like the pedagogical agent, demonstrations, or the narrative. Next, the interaction between player-bound and game-bound mechanics creates learning affordances. These allow players, through repetition, to form mental models and apply their knowledge.
• Figure 2: Interaction and data annotation mechanic of the Publish game mode (Section \ref{['sec:gamemodes:details']}). First, players tap biased words. Then, they swipe or use the buttons to annotate the sentence as "biased" or "not biased."
• Figure 3: Direct and delayed feedback in the Publish game mode (Section \ref{['sec:gamemodes:details']}). Sentence level feedback color-codes the card. In this case, it turns green and indicates a hit. In case of a miss, it turns red. Next, correctly and incorrectly annotated words are shown in green and red. Missed words display a black outline. Stopwords do not count in the feedback and are displayed as right when surrounded by biased words. The third card shows delayed feedback. If a ground truth has not been established, sentence and word level feedback is displayed in yellow.
• Figure 4: The first screen (left to right) shows the detailed feedback for one correctly annotated sentence in the Paper section. It displays the sentence card with one word highlighted in red (incorrect annotation), one in green (correct annotation), and one with a black outline (missed word). Players can collect the reward or navigate to the discussion. The second screen shows the discussion of a sentence between two players. The sentence card shows on top with the comments below. The third screen shows the shop with six unlockable topics.
• Figure 5: The first left screen shows feedback in the game mode Context, with a sentence card displayed. It includes a green hook to indicate a correct answer on top, and the plant speaks motivationally in the left corner. The middle screen displays the topic selection before Publish. One topic was already played (black button with yellow refill button), and three are playable (white buttons). The right screen displays a Quick Words screen with the sentence card, feedback (green highlight for correct answer, red highlight for incorrect answer, yellow highlight for delayed feedback), the time bar, and the button for the next sentence in the bottom right corner.