SocialNLP Fake-EmoReact 2021 Challenge Overview: Predicting Fake Tweets from Their Replies and GIFs

TL;DR

The paper introduces the Fake-EmoReact 2021 shared task, which targets detecting fake tweets using reply context and GIF-based reactions. It provides a large dataset of over 453k tweets with labeled authenticity and GIF categories, distributed across two evaluation rounds with macro-F1 as the metric. The results show strong performance from transformer-based models with voting and from traditional ML with careful feature handling, with the top team achieving a macro-F1 of 93.9%. This work demonstrates the viability of contextual and visual-reaction cues for misinformation detection and offers a valuable resource for future research, including dealing with imbalanced reaction categories.

Abstract

This paper provides an overview of the Fake-EmoReact 2021 Challenge, held at the 9th SocialNLP Workshop, in conjunction with NAACL 2021. The challenge requires predicting the authenticity of tweets using reply context and augmented GIF categories from EmotionGIF dataset. We offer the Fake-EmoReact dataset with more than 453k as the experimental materials, where every tweet is labeled with authenticity. Twenty-four teams registered to participate in this challenge, and 5 submitted their results successfully in the evaluation phase. The best team achieves 93.9 on Fake-EmoReact 2021 dataset using F1 score. In addition, we show the definition of share task, data collection, and the teams' performance that joined this challenge and their approaches.

Figures (1)

• Figure 1: An example of fake new message on Twitter