Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Coreference Resolution for Vietnamese Narrative Texts

Hieu-Dai Tran, Duc-Vu Nguyen, Ngan Luu-Thuy Nguyen

TL;DR

The paper tackles coreference resolution in Vietnamese narrative texts, a low-resource setting, by building a manually annotated dataset from VnExpress, establishing explicit annotation guidelines, and employing few-shot prompting with GPT-3.5-Turbo and GPT-4. It evaluates model performance using CoNLL F1, comprising MUC, B-Cubed, and CEAF_phi, and finds GPT-4 significantly superior in both accuracy and response consistency. Case studies illustrate GPT-4’s ability to correctly cluster speakers, differentiate entities, and handle complex noun phrases. The work provides a valuable Vietnamese coreference resource, demonstrates the viability of advanced LLMs for this task, and outlines future directions for data expansion, domain adaptation, and efficiency improvements.

Abstract

Coreference resolution is a vital task in natural language processing (NLP) that involves identifying and linking different expressions in a text that refer to the same entity. This task is particularly challenging for Vietnamese, a low-resource language with limited annotated datasets. To address these challenges, we developed a comprehensive annotated dataset using narrative texts from VnExpress, a widely-read Vietnamese online news platform. We established detailed guidelines for annotating entities, focusing on ensuring consistency and accuracy. Additionally, we evaluated the performance of large language models (LLMs), specifically GPT-3.5-Turbo and GPT-4, on this dataset. Our results demonstrate that GPT-4 significantly outperforms GPT-3.5-Turbo in terms of both accuracy and response consistency, making it a more reliable tool for coreference resolution in Vietnamese.

Coreference Resolution for Vietnamese Narrative Texts

TL;DR

The paper tackles coreference resolution in Vietnamese narrative texts, a low-resource setting, by building a manually annotated dataset from VnExpress, establishing explicit annotation guidelines, and employing few-shot prompting with GPT-3.5-Turbo and GPT-4. It evaluates model performance using CoNLL F1, comprising MUC, B-Cubed, and CEAF_phi, and finds GPT-4 significantly superior in both accuracy and response consistency. Case studies illustrate GPT-4’s ability to correctly cluster speakers, differentiate entities, and handle complex noun phrases. The work provides a valuable Vietnamese coreference resource, demonstrates the viability of advanced LLMs for this task, and outlines future directions for data expansion, domain adaptation, and efficiency improvements.

Abstract

Coreference resolution is a vital task in natural language processing (NLP) that involves identifying and linking different expressions in a text that refer to the same entity. This task is particularly challenging for Vietnamese, a low-resource language with limited annotated datasets. To address these challenges, we developed a comprehensive annotated dataset using narrative texts from VnExpress, a widely-read Vietnamese online news platform. We established detailed guidelines for annotating entities, focusing on ensuring consistency and accuracy. Additionally, we evaluated the performance of large language models (LLMs), specifically GPT-3.5-Turbo and GPT-4, on this dataset. Our results demonstrate that GPT-4 significantly outperforms GPT-3.5-Turbo in terms of both accuracy and response consistency, making it a more reliable tool for coreference resolution in Vietnamese.
Paper Structure (14 sections, 2 figures, 2 tables)

This paper contains 14 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Dataset Construction
  3. Collecting Narrative Texts
  4. Annotation Guidelines
  5. Tool for Annotation
  6. Definition of Entities
  7. Prompting
  8. Evaluation
  9. Evaluation Metrics
  10. Results
  11. Case Study
  12. Discussion
  13. Conclusion
  14. Future Work

Figures (2)

  • Figure 1: The process of generating mention clusters from raw text using LLMs. The input document is processed to return a list of tuples containing tags of mentions referring to the same entity. For example, in the input text, tags identify various entities and group them into clusters, as shown in the output.
  • Figure 2: User interface of the Coreference Annotation Tool with SACR