Malaysian English News Decoded: A Linguistic Resource for Named Entity and Relation Extraction

TL;DR

The paper tackles the gap in NLP resources for Malaysian English (ME) by showing standard English datasets inadequately support ME tasks, particularly in NER and RE. It presents MEN-Dataset, a manually annotated collection of 200 Malaysian English news articles with 6,061 entities and 3,268 relation instances, including ME-adapted labels TITLE and ROLE, and provides rigorous inter-annotator agreement analysis. The authors fine-tune spaCy NER models on MEN-Dataset, finding that a blank model fine-tuned on the dataset achieves the best performance (F1 ≈ 0.94), with substantial average gains (~230%) over baseline models. They release the dataset and annotation guidelines on GitHub, offering a valuable resource to advance ME NLP research, particularly for NER and relation extraction in low-resource, linguistically diverse contexts.

Abstract

Standard English and Malaysian English exhibit notable differences, posing challenges for natural language processing (NLP) tasks on Malaysian English. Unfortunately, most of the existing datasets are mainly based on standard English and therefore inadequate for improving NLP tasks in Malaysian English. An experiment using state-of-the-art Named Entity Recognition (NER) solutions on Malaysian English news articles highlights that they cannot handle morphosyntactic variations in Malaysian English. To the best of our knowledge, there is no annotated dataset available to improvise the model. To address these issues, we constructed a Malaysian English News (MEN) dataset, which contains 200 news articles that are manually annotated with entities and relations. We then fine-tuned the spaCy NER tool and validated that having a dataset tailor-made for Malaysian English could improve the performance of NER in Malaysian English significantly. This paper presents our effort in the data acquisition, annotation methodology, and thorough analysis of the annotated dataset. To validate the quality of the annotation, inter-annotator agreement was used, followed by adjudication of disagreements by a subject matter expert. Upon completion of these tasks, we managed to develop a dataset with 6,061 entities and 3,268 relation instances. Finally, we discuss on spaCy fine-tuning setup and analysis on the NER performance. This unique dataset will contribute significantly to the advancement of NLP research in Malaysian English, allowing researchers to accelerate their progress, particularly in NER and relation extraction. The dataset and annotation guideline has been published on Github.

Figures (5)

• Figure 1: Phases in the annotation process to annotate news article for each milestone. This phase has helped to ensure accuracy and consistency.
• Figure 2: Snippet of news article that has been annotated with entities and relations. The annotated entities are underlined, and their corresponding entity labels are included below the line. Additionally, we linked the relations between two annotated entities. Each relation has a suffix of the dataset name, which indicates the dataset from which the relation label has been adapted.
• Figure 3: F1-Score calculated to measure the agreement based on entity labels.
• Figure 4: Total and Unique Entity Mention Annotated based on the Labels.
• Figure 5: Proccessing steps in spaCy pipeline, it starts by giving the input. The outcome will include linguistic annotations after all the processing steps.