Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CMNER: A Chinese Multimodal NER Dataset based on Social Media

Yuanze Ji, Bobo Li, Jun Zhou, Fei Li, Chong Teng, Donghong Ji

TL;DR

CMNER presents a new Chinese multimodal NER dataset from Weibo, addressing data scarcity by pairing 5,000 posts with 18,326 images across PER, LOC, ORG, and MISC. It establishes strong baselines with ACN and UMT models, showing that visual information enhances NER performance, and demonstrates the value of one-text-multi-image data. The paper further explores cross-lingual transfer with Twitter2015, using translation-based data augmentation to reveal mutual improvements between Chinese and English multimodal NER. The findings highlight the dataset's quality and the potential of cross-lingual multimodal signals to advance NER in multilingual settings, with public release of CMNER for future research.

Abstract

Multimodal Named Entity Recognition (MNER) is a pivotal task designed to extract named entities from text with the support of pertinent images. Nonetheless, a notable paucity of data for Chinese MNER has considerably impeded the progress of this natural language processing task within the Chinese domain. Consequently, in this study, we compile a Chinese Multimodal NER dataset (CMNER) utilizing data sourced from Weibo, China's largest social media platform. Our dataset encompasses 5,000 Weibo posts paired with 18,326 corresponding images. The entities are classified into four distinct categories: person, location, organization, and miscellaneous. We perform baseline experiments on CMNER, and the outcomes underscore the effectiveness of incorporating images for NER. Furthermore, we conduct cross-lingual experiments on the publicly available English MNER dataset (Twitter2015), and the results substantiate our hypothesis that Chinese and English multimodal NER data can mutually enhance the performance of the NER model.

CMNER: A Chinese Multimodal NER Dataset based on Social Media

TL;DR

CMNER presents a new Chinese multimodal NER dataset from Weibo, addressing data scarcity by pairing 5,000 posts with 18,326 images across PER, LOC, ORG, and MISC. It establishes strong baselines with ACN and UMT models, showing that visual information enhances NER performance, and demonstrates the value of one-text-multi-image data. The paper further explores cross-lingual transfer with Twitter2015, using translation-based data augmentation to reveal mutual improvements between Chinese and English multimodal NER. The findings highlight the dataset's quality and the potential of cross-lingual multimodal signals to advance NER in multilingual settings, with public release of CMNER for future research.

Abstract

Multimodal Named Entity Recognition (MNER) is a pivotal task designed to extract named entities from text with the support of pertinent images. Nonetheless, a notable paucity of data for Chinese MNER has considerably impeded the progress of this natural language processing task within the Chinese domain. Consequently, in this study, we compile a Chinese Multimodal NER dataset (CMNER) utilizing data sourced from Weibo, China's largest social media platform. Our dataset encompasses 5,000 Weibo posts paired with 18,326 corresponding images. The entities are classified into four distinct categories: person, location, organization, and miscellaneous. We perform baseline experiments on CMNER, and the outcomes underscore the effectiveness of incorporating images for NER. Furthermore, we conduct cross-lingual experiments on the publicly available English MNER dataset (Twitter2015), and the results substantiate our hypothesis that Chinese and English multimodal NER data can mutually enhance the performance of the NER model.
Paper Structure (24 sections, 4 figures, 4 tables)

This paper contains 24 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NER
  4. Multimodal NER
  5. Cross-lingual Transfer
  6. Dataset Construction
  7. Dataset Design
  8. Preparation and Annotation
  9. Data Statistics
  10. Baseline Experiments
  11. Task Formulation
  12. Model Architectures
  13. Experiment Setup
  14. Results
  15. Analysis
  16. ...and 9 more sections

Figures (4)

  • Figure 1: An example of multimodal Weibo post.
  • Figure 2: Two baseline models in the experiment. (a) is the architecture of ACN and (b) is the architecture of UMT.
  • Figure 3: Examples of MNER. The "gold" is the golden entity type in the text and "w/o pics" means without visual inputs. "√" denotes the prediction is completely right, "×" denotes the prediction is totally wrong, and "$\tilde{√}$" denotes the prediction is incomplete right.
  • Figure 4: Cases of different error types.