Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

Gaurish Thakkar, Sherzod Hakimov, Marko Tadić

Abstract

In recent years, multimodal natural language processing, aimed at learning from diverse data types, has garnered significant attention. However, there needs to be more clarity when it comes to analysing multimodal tasks in multi-lingual contexts. While prior studies on sentiment analysis of tweets have predominantly focused on the English language, this paper addresses this gap by transforming an existing textual Twitter sentiment dataset into a multimodal format through a straightforward curation process. Our work opens up new avenues for sentiment-related research within the research community. Additionally, we conduct baseline experiments utilising this augmented dataset and report the findings. Notably, our evaluations reveal that when comparing unimodal and multimodal configurations, using a sentiment-tuned large language model as a text encoder performs exceptionally well.

M2SA: Multimodal and Multilingual Model for Sentiment Analysis of Tweets

Abstract

In recent years, multimodal natural language processing, aimed at learning from diverse data types, has garnered significant attention. However, there needs to be more clarity when it comes to analysing multimodal tasks in multi-lingual contexts. While prior studies on sentiment analysis of tweets have predominantly focused on the English language, this paper addresses this gap by transforming an existing textual Twitter sentiment dataset into a multimodal format through a straightforward curation process. Our work opens up new avenues for sentiment-related research within the research community. Additionally, we conduct baseline experiments utilising this augmented dataset and report the findings. Notably, our evaluations reveal that when comparing unimodal and multimodal configurations, using a sentiment-tuned large language model as a text encoder performs exceptionally well.
Paper Structure (23 sections, 4 figures, 2 tables)

This paper contains 23 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Multilingual Sentiment Analysis (M2SA)
  4. Data Collection
  5. Preprocessing
  6. Dataset
  7. Methodology
  8. Problem Definition
  9. Text Encoders
  10. Multilingual-BERT (M-BERT)
  11. XLM-RoBERTa (XLM-R)
  12. XLM-RoBERTa-Sentiment-Multilingual (XLMR-SM)
  13. Vision Encoders
  14. CLIP
  15. DINOv2
  16. ...and 8 more sections

Figures (4)

  • Figure 1: The dataset's distribution across different languages.
  • Figure 2: Model architecture of the M2SA.
  • Figure 3: The (left) plot illustrates the averaged F1-score across various models. The (right) plot illustrates the averaged F1-score across languages.
  • Figure 4: Examples from multilingual multimodal model predicts the correct label and text-only model fail