Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages

Weerayut Buaphet, Thanh-Nhi Nguyen, Risa Kondo, Tomoyuki Kajiwara, Yumin Kim, Jimin Lee, Hwanhee Lee, Holy Lovenia, Peerat Limkonchotiwat, Sarana Nutanong, Rob Van der Goot

TL;DR

This work addresses the limited cross-language coverage of lexical normalization benchmarks by extending MultiLexNorm to five Asian languages across four scripts (MultiLexNorm++). It introduces an LLM-enabled normalization pipeline that combines span detection, a training-data lookup table, and in-context prompting, evaluated across multiple open and proprietary models. Results show that non-Latin-script languages lag behind Latin-script ones, with GPT-4o achieving parity with, or superiority over, the established UFAL baseline on several languages, while still facing challenges in Japanese and Korean. The study highlights practical considerations, including detection accuracy, segmentation effects, and cost, and provides a roadmap for robust, multilingual lexical normalization with potential downstream benefits for NLP tasks in diverse language communities.

Abstract

Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian languages from different language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task.

MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages

TL;DR

This work addresses the limited cross-language coverage of lexical normalization benchmarks by extending MultiLexNorm to five Asian languages across four scripts (MultiLexNorm++). It introduces an LLM-enabled normalization pipeline that combines span detection, a training-data lookup table, and in-context prompting, evaluated across multiple open and proprietary models. Results show that non-Latin-script languages lag behind Latin-script ones, with GPT-4o achieving parity with, or superiority over, the established UFAL baseline on several languages, while still facing challenges in Japanese and Korean. The study highlights practical considerations, including detection accuracy, segmentation effects, and cost, and provides a roadmap for robust, multilingual lexical normalization with potential downstream benefits for NLP tasks in diverse language communities.

Abstract

Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian languages from different language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task.
Paper Structure (29 sections, 5 figures, 9 tables)

This paper contains 29 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MultiLexNorm++
  4. Languages
  5. Indonesian
  6. Japanese
  7. Korean
  8. Thai
  9. Vietnamese
  10. Evaluation Setup
  11. Baselines
  12. LLM-based Pipeline
  13. Results
  14. Analysis
  15. Finding the right normalization
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Normalization examples across languages. A shows the original informal or non-standard text, with A (prime) providing its romanized transliteration. B shows the normalized form, with B (prime) its romanized transliteration. English translations are provided in square brackets.
  • Figure 2: Two-shot In-Context Learning: ①Detection: An encoder-based model detects typos in informative words. ②Dictionary: dictionary lookup with Miller-Madow entropy selection ③LLM: in-context prompt prediction with an LLM.
  • Figure 3: Normalized prediction distributions across languages for each LLM.
  • Figure 4: Error rates normalized by the maximum count in each LLM error category.
  • Figure 5: Varying in-context size based on the development set