Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing Approaches to Automatic Summarization in Less-Resourced Languages

Chester Palen-Michel, Constantine Lignos

TL;DR

The paper addresses automatic summarization in less-resourced languages by systematically comparing zero-shot prompting of small and large LLMs, fine-tuning multilingual models like mT5 with and without data augmentation, and translation-based pipelines. It finds that multilingual transfer with fine-tuned mT5 often yields the strongest performance across metrics, while larger LLMs and translation pipelines do not consistently surpass the multilingual baseline, and LLM-based evaluation (M-Prometheus) may be less reliable for low-resource languages. Data augmentation helps when fine-tuning individual languages but does not universally outperform multilingual fine-tuning, and LLMs frequently generate English or commentary rather than staying in the target language. The study highlights the importance of robust, multilingual evaluation and openness to collaborative, language-community-led NLP efforts, concluding with a public release of generated summaries to support future benchmarking and evaluation.

Abstract

Automatic text summarization has achieved high performance in high-resourced languages like English, but comparatively less attention has been given to summarization in less-resourced languages. This work compares a variety of different approaches to summarization from zero-shot prompting of LLMs large and small to fine-tuning smaller models like mT5 with and without three data augmentation approaches and multilingual transfer. We also explore an LLM translation pipeline approach, translating from the source language to English, summarizing and translating back. Evaluating with five different metrics, we find that there is variation across LLMs in their performance across similar parameter sizes, that our multilingual fine-tuned mT5 baseline outperforms most other approaches including zero-shot LLM performance for most metrics, and that LLM as judge may be less reliable on less-resourced languages.

Comparing Approaches to Automatic Summarization in Less-Resourced Languages

TL;DR

The paper addresses automatic summarization in less-resourced languages by systematically comparing zero-shot prompting of small and large LLMs, fine-tuning multilingual models like mT5 with and without data augmentation, and translation-based pipelines. It finds that multilingual transfer with fine-tuned mT5 often yields the strongest performance across metrics, while larger LLMs and translation pipelines do not consistently surpass the multilingual baseline, and LLM-based evaluation (M-Prometheus) may be less reliable for low-resource languages. Data augmentation helps when fine-tuning individual languages but does not universally outperform multilingual fine-tuning, and LLMs frequently generate English or commentary rather than staying in the target language. The study highlights the importance of robust, multilingual evaluation and openness to collaborative, language-community-led NLP efforts, concluding with a public release of generated summaries to support future benchmarking and evaluation.

Abstract

Automatic text summarization has achieved high performance in high-resourced languages like English, but comparatively less attention has been given to summarization in less-resourced languages. This work compares a variety of different approaches to summarization from zero-shot prompting of LLMs large and small to fine-tuning smaller models like mT5 with and without three data augmentation approaches and multilingual transfer. We also explore an LLM translation pipeline approach, translating from the source language to English, summarizing and translating back. Evaluating with five different metrics, we find that there is variation across LLMs in their performance across similar parameter sizes, that our multilingual fine-tuned mT5 baseline outperforms most other approaches including zero-shot LLM performance for most metrics, and that LLM as judge may be less reliable on less-resourced languages.
Paper Structure (39 sections, 8 figures, 15 tables)

This paper contains 39 sections, 8 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Summarization Evaluation
  5. Datasets
  6. Methodology
  7. Fine-tuning mT5 with Data Augmentation
  8. Extractive-Training
  9. Self-Training
  10. Back-Summarization
  11. Individual Models with Augmented Data
  12. Multilingual Models with Augmented Data
  13. LLM Experiments
  14. Smaller LLMs
  15. Results
  16. ...and 24 more sections

Figures (8)

  • Figure 1: Methodology for generating additional training examples from Wikipedia articles
  • Figure 2: Scores for augmentation approaches with individual-language fine-tuning using mT5. Aya-101 performance is added for comparison with LLM performance.
  • Figure 3: Scores for augmentation approaches with multilingual fine-tuning by language
  • Figure 4: Comparison of Aya-101, Aya-Expanse, and Multilingual Transfer MT5 finetuned baseline
  • Figure 5: Proportion of summaries with extra English text generated for each model and approach by language by category of Larger LLMs and Translate-Summarize-Translate approach.
  • ...and 3 more figures