Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication

Hanjia Lyu, Weihong Qi, Zhongyu Wei, Jiebo Luo

TL;DR

This work investigates how GPT-4V interprets and uses emojis compared with humans in social-media contexts. It conducts two studies: first, comparing GPT-4V's single-word emoji descriptions to human annotations to assess interpretive alignment; second, prompting GPT-4V to generate three emojis for TikTok-like contexts and comparing usage patterns via embedding analyses. The results reveal category-dependent interpretive gaps and a mixture of human-like and divergent emoji usage, with notable biases likely stemming from English-centric training and cultural representation gaps. These findings highlight both the strengths and limitations of LMMs in navigating symbolic digital language and underscore the need for more culturally diverse data to improve emoji understanding in AI systems.

Abstract

Leveraging Large Multimodal Models (LMMs) to simulate human behaviors when processing multimodal information, especially in the context of social media, has garnered immense interest due to its broad potential and far-reaching implications. Emojis, as one of the most unique aspects of digital communication, are pivotal in enriching and often clarifying the emotional and tonal dimensions. Yet, there is a notable gap in understanding how these advanced models, such as GPT-4V, interpret and employ emojis in the nuanced context of online interaction. This study intends to bridge this gap by examining the behavior of GPT-4V in replicating human-like use of emojis. The findings reveal a discernible discrepancy between human and GPT-4V behaviors, likely due to the subjective nature of human interpretation and the limitations of GPT-4V's English-centric training, suggesting cultural biases and inadequate representation of non-English cultures.

Human vs. LMMs: Exploring the Discrepancy in Emoji Interpretation and Usage in Digital Communication

TL;DR

This work investigates how GPT-4V interprets and uses emojis compared with humans in social-media contexts. It conducts two studies: first, comparing GPT-4V's single-word emoji descriptions to human annotations to assess interpretive alignment; second, prompting GPT-4V to generate three emojis for TikTok-like contexts and comparing usage patterns via embedding analyses. The results reveal category-dependent interpretive gaps and a mixture of human-like and divergent emoji usage, with notable biases likely stemming from English-centric training and cultural representation gaps. These findings highlight both the strengths and limitations of LMMs in navigating symbolic digital language and underscore the need for more culturally diverse data to improve emoji understanding in AI systems.

Abstract

Leveraging Large Multimodal Models (LMMs) to simulate human behaviors when processing multimodal information, especially in the context of social media, has garnered immense interest due to its broad potential and far-reaching implications. Emojis, as one of the most unique aspects of digital communication, are pivotal in enriching and often clarifying the emotional and tonal dimensions. Yet, there is a notable gap in understanding how these advanced models, such as GPT-4V, interpret and employ emojis in the nuanced context of online interaction. This study intends to bridge this gap by examining the behavior of GPT-4V in replicating human-like use of emojis. The findings reveal a discernible discrepancy between human and GPT-4V behaviors, likely due to the subjective nature of human interpretation and the limitations of GPT-4V's English-centric training, suggesting cultural biases and inadequate representation of non-English cultures.
Paper Structure (22 sections, 1 equation, 4 figures, 3 tables)

This paper contains 22 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiments
  4. Study 1: Emoji Interpretation
  5. Study Design
  6. Emoji Selection
  7. Prompting GPT-4V
  8. Human Annotations
  9. Results
  10. Ambiguity Analysis
  11. Study 2: Emoji Usage
  12. Study Design
  13. Prompting GPT-4V
  14. Context
  15. Data Collection
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Variations in semantic interpretation between GPT-4V and humans for emojis across different categories.
  • Figure 2: Comparison of the degree of ambiguity in emoji interpretation between humans and GPT-4V across different emoji categories, along with the symbolicalness rating of these categories. The red diagonal line represents the point of equality where GPT-4V and humans exhibit the same level of ambiguity in emoji interpretation.
  • Figure 3: PCA visualization word embeddings of human-used and GPT-4V generated emojis.
  • Figure 4: PCA visualization word embeddings of human-used and GPT-4V generated emojis (additional data).