Think Carefully and Check Again! Meta-Generation Unlocking LLMs for Low-Resource Cross-Lingual Summarization

TL;DR

The paper tackles cross-lingual summarization for low-resource languages by introducing SITR, a four-step zero-shot framework (Summarization, Improvement, Translation, Refinement) that employs two-stage meta-generation to iteratively refine outputs. By prompting LLMs with task-specific instructions, SITR enables both high-resource and open-source models, notably GPT-3.5 and GPT-4, to surpass baselines and even fine-tuned models on CrossSum and WikiLingua. The results demonstrate significant improvements in both lexical and semantic quality (ROUGE and BERTScore) and reveal the robustness of SITR across models and languages, including open-source LLMs like Llama3 and Gemma. Overall, the method provides a scalable path to high-quality cross-lingual summaries for languages with limited data, expanding the practical reach of LLM-enabled CLS.

Abstract

Cross-lingual summarization (CLS) aims to generate a summary for the source text in a different target language. Currently, instruction-tuned large language models (LLMs) excel at various English tasks. However, unlike languages such as English, Chinese or Spanish, for those relatively low-resource languages with limited usage or data, recent studies have shown that LLMs' performance on CLS tasks remains unsatisfactory even with few-shot settings. This raises the question: Are LLMs capable of handling cross-lingual summarization tasks for low-resource languages? To resolve this question, we fully explore the potential of large language models on cross-lingual summarization task for low-resource languages through our four-step zero-shot method: Summarization, Improvement, Translation and Refinement (SITR) with correspondingly designed prompts. We test our proposed method with multiple LLMs on two well-known cross-lingual summarization datasets with various low-resource target languages. The results show that: i) GPT-3.5 and GPT-4 significantly and consistently outperform other baselines when using our zero-shot SITR methods. ii) By employing our proposed method, we unlock the potential of LLMs, enabling them to effectively handle cross-lingual summarization tasks for relatively low-resource languages.

Figures (12)

• Figure 1: An example of single-step summarize-translate method for cross-lingual summarization.
• Figure 2: The architecture of our four-step zero-shot SITR method for cross-lingual summarization.
• Figure 3: Comparison of three different LLM methods on one single test example to summarize English source text in Ukarainian. The English translation of each model output is shown in brackets.
• Figure 4: Comparison of the BERTScore after removing key meta-generation steps.
• Figure 5: Comparison of the sum of ROUGE-1/2/L after removing key meta-generation steps.