Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition

Hussain Ahmad, Qingyang Zeng, Jing Wan

TL;DR

Urdu MNER faces data scarcity and linguistic complexity, motivating the authors to introduce Twitter2015-Urdu, the first Urdu multimodal NER dataset adapted from Twitter2015, and the U-MNER framework. U-MNER fuses Urdu-BERT-based text representations with ResNet-derived visual features through a cross-modal fusion module and a Visual Gate, feeding a BiLSTM-CRF decoder to produce BIO labels for text augmented by image context. The work provides comprehensive baselines for Urdu MNER (text-only and multimodal) and demonstrates state-of-the-art performance (F1 up to 62.75%) on Twitter2015-Urdu, with extensive ablations and case analyses validating the proposed cross-modal design. By releasing both the dataset and the framework, the authors enable robust multimodal NER research for a key low-resource language and set a foundation for broader social media analysis in Urdu-speaking contexts.

Abstract

The emergence of multimodal content, particularly text and images on social media, has positioned Multimodal Named Entity Recognition (MNER) as an increasingly important area of research within Natural Language Processing. Despite progress in high-resource languages such as English, MNER remains underexplored for low-resource languages like Urdu. The primary challenges include the scarcity of annotated multimodal datasets and the lack of standardized baselines. To address these challenges, we introduce the U-MNER framework and release the Twitter2015-Urdu dataset, a pioneering resource for Urdu MNER. Adapted from the widely used Twitter2015 dataset, it is annotated with Urdu-specific grammar rules. We establish benchmark baselines by evaluating both text-based and multimodal models on this dataset, providing comparative analyses to support future research on Urdu MNER. The U-MNER framework integrates textual and visual context using Urdu-BERT for text embeddings and ResNet for visual feature extraction, with a Cross-Modal Fusion Module to align and fuse information. Our model achieves state-of-the-art performance on the Twitter2015-Urdu dataset, laying the groundwork for further MNER research in low-resource languages.

A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition

TL;DR

Urdu MNER faces data scarcity and linguistic complexity, motivating the authors to introduce Twitter2015-Urdu, the first Urdu multimodal NER dataset adapted from Twitter2015, and the U-MNER framework. U-MNER fuses Urdu-BERT-based text representations with ResNet-derived visual features through a cross-modal fusion module and a Visual Gate, feeding a BiLSTM-CRF decoder to produce BIO labels for text augmented by image context. The work provides comprehensive baselines for Urdu MNER (text-only and multimodal) and demonstrates state-of-the-art performance (F1 up to 62.75%) on Twitter2015-Urdu, with extensive ablations and case analyses validating the proposed cross-modal design. By releasing both the dataset and the framework, the authors enable robust multimodal NER research for a key low-resource language and set a foundation for broader social media analysis in Urdu-speaking contexts.

Abstract

The emergence of multimodal content, particularly text and images on social media, has positioned Multimodal Named Entity Recognition (MNER) as an increasingly important area of research within Natural Language Processing. Despite progress in high-resource languages such as English, MNER remains underexplored for low-resource languages like Urdu. The primary challenges include the scarcity of annotated multimodal datasets and the lack of standardized baselines. To address these challenges, we introduce the U-MNER framework and release the Twitter2015-Urdu dataset, a pioneering resource for Urdu MNER. Adapted from the widely used Twitter2015 dataset, it is annotated with Urdu-specific grammar rules. We establish benchmark baselines by evaluating both text-based and multimodal models on this dataset, providing comparative analyses to support future research on Urdu MNER. The U-MNER framework integrates textual and visual context using Urdu-BERT for text embeddings and ResNet for visual feature extraction, with a Cross-Modal Fusion Module to align and fuse information. Our model achieves state-of-the-art performance on the Twitter2015-Urdu dataset, laying the groundwork for further MNER research in low-resource languages.
Paper Structure (48 sections, 14 equations, 5 figures, 5 tables)

This paper contains 48 sections, 14 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. RELATED WORK
  3. Multimodal Named Entity Recognition
  4. Challenges in Urdu Named Entity Recognition
  5. Urdu MNER Dataset
  6. CORPUS CREATION
  7. Data Collection
  8. Data Preprocessing
  9. Tokenized Data Conversion
  10. Data Cleaning
  11. Normalization
  12. Translation & Review
  13. Automated Translation
  14. Human Review
  15. Cultural Adaptation and Transliterations
  16. ...and 33 more sections

Figures (5)

  • Figure 1: An example of Multimodal Named Entity Recognition, with the named entity and its type highlighted in brackets.
  • Figure 2: Dataset development pipeline.
  • Figure 3: Overall Architecture of U-MNER Framework.
  • Figure 4: Case study illustrating how visual data improves entity recognition by resolving ambiguities and enhancing classification in multimodal NER.
  • Figure 5: Error analysis examples from U-MNER, showing misclassifications due to image-text mismatch, contextual ambiguity, and challenges with abbreviations.