Image Matters: A New Dataset and Empirical Study for Multimodal Hyperbole Detection

TL;DR

A multimodal hyperbole detection dataset from Weibo is created and some studies on it are carried out, which explores the role of text and image for hyperbole detection and evaluates the cross-domain performance of different models.

Abstract

Hyperbole, or exaggeration, is a common linguistic phenomenon. The detection of hyperbole is an important part of understanding human expression. There have been several studies on hyperbole detection, but most of which focus on text modality only. However, with the development of social media, people can create hyperbolic expressions with various modalities, including text, images, videos, etc. In this paper, we focus on multimodal hyperbole detection. We create a multimodal detection dataset from Weibo (a Chinese social media) and carry out some studies on it. We treat the text and image from a piece of weibo as two modalities and explore the role of text and image for hyperbole detection. Different pre-trained multimodal encoders are also evaluated on this downstream task to show their performance. Besides, since this dataset is constructed from five different topics, we also evaluate the cross-domain performance of different models. These studies can serve as a benchmark and point out the direction of further study on multimodal hyperbole detection.

Figures (7)

• Figure 1: Examples of image aiding the detection of hyperbole or serving as background information(all the examples are hyperbole). (a). In the image we can see that it is not so cold. (This is the surface of a weather application which says it is $13^\circ C$ in Urumqi. It's much colder in winter in Urumqi.) So we must refer to both text and image to correctly determine the post as hyperbole.; (b).The image is necessary because the text just states some fact. But with the melting face we can confidently label it as hyperbole because however hot it is, people do not melt.; (c). The grey background and stiff face in the image increases the confidence of annotating it as hyperbole.;(d).The image is just a screenshot of a navigating software. The hyperbolic meaning here comes from that people do not die from just going to this scenic spot, which cannot be revealed by this image.
• Figure 2: An example of hyperbolic post expressing their love towards a star with the word 风景(scenery).(There are many much longer examples, for conciseness we choose a relatively short one here.)
• Figure 3: An example of images with hyperbolic words in it. (The words in the images say "I hate the whole world before sleep.")
• Figure 4: The architecture of the attn-gate fusion model.
• Figure 5: Two examples of hyperbole cases.