Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of Many-to-Many Summarization with Large Language Models

Jiaan Wang, Fandong Meng, Zengkui Sun, Yunlong Liang, Yuxuan Cao, Jiarong Xu, Haoxiang Shi, Jie Zhou

TL;DR

This paper systematically evaluates many-to-many summarization (M2MS) with large language models (LLMs) by constructing a 47.8K-sample, multi-domain dataset from eight multilingual sources and benchmarking 18 LLMs under zero-shot and instruction-tuning regimes, with traditional baselines for comparison. The results show zero-shot LLMs can match fine-tuned baselines, while instruction-tuned open-source LLMs significantly outperform both zero-shot LLMs and traditional models on automatic metrics, sometimes rivaling or surpassing GPT-4o. Importantly, instruction-tuning does not degrade general abilities like MMLU, but human evaluations reveal persistent factuality issues, with tuning potentially increasing hallucinations. The work highlights practical considerations for deploying LLM-based M2MS, including long-document handling and the need for methods to control factual errors in real applications.

Abstract

Many-to-many summarization (M2MS) aims to process documents in any language and generate the corresponding summaries also in any language. Recently, large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform M2MS in real applications. This work presents a systematic empirical study on LLMs' M2MS ability. Specifically, we first reorganize M2MS data based on eight previous domain-specific datasets. The reorganized data contains 47.8K samples spanning five domains and six languages, which could be used to train and evaluate LLMs. Then, we benchmark 18 LLMs in a zero-shot manner and an instruction-tuning manner. Fine-tuned traditional models (e.g., mBART) are also conducted for comparisons. Our experiments reveal that, zero-shot LLMs achieve competitive results with fine-tuned traditional models. After instruct-tuning, open-source LLMs can significantly improve their M2MS ability, and outperform zero-shot LLMs (including GPT-4) in terms of automatic evaluations. In addition, we demonstrate that this task-specific improvement does not sacrifice the LLMs' general task-solving abilities. However, as revealed by our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify the issue. Thus, how to control factual errors becomes the key when building LLM summarizers in real applications, and is worth noting in future research.

An Empirical Study of Many-to-Many Summarization with Large Language Models

TL;DR

This paper systematically evaluates many-to-many summarization (M2MS) with large language models (LLMs) by constructing a 47.8K-sample, multi-domain dataset from eight multilingual sources and benchmarking 18 LLMs under zero-shot and instruction-tuning regimes, with traditional baselines for comparison. The results show zero-shot LLMs can match fine-tuned baselines, while instruction-tuned open-source LLMs significantly outperform both zero-shot LLMs and traditional models on automatic metrics, sometimes rivaling or surpassing GPT-4o. Importantly, instruction-tuning does not degrade general abilities like MMLU, but human evaluations reveal persistent factuality issues, with tuning potentially increasing hallucinations. The work highlights practical considerations for deploying LLM-based M2MS, including long-document handling and the need for methods to control factual errors in real applications.

Abstract

Many-to-many summarization (M2MS) aims to process documents in any language and generate the corresponding summaries also in any language. Recently, large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform M2MS in real applications. This work presents a systematic empirical study on LLMs' M2MS ability. Specifically, we first reorganize M2MS data based on eight previous domain-specific datasets. The reorganized data contains 47.8K samples spanning five domains and six languages, which could be used to train and evaluate LLMs. Then, we benchmark 18 LLMs in a zero-shot manner and an instruction-tuning manner. Fine-tuned traditional models (e.g., mBART) are also conducted for comparisons. Our experiments reveal that, zero-shot LLMs achieve competitive results with fine-tuned traditional models. After instruct-tuning, open-source LLMs can significantly improve their M2MS ability, and outperform zero-shot LLMs (including GPT-4) in terms of automatic evaluations. In addition, we demonstrate that this task-specific improvement does not sacrifice the LLMs' general task-solving abilities. However, as revealed by our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify the issue. Thus, how to control factual errors becomes the key when building LLM summarizers in real applications, and is worth noting in future research.
Paper Structure (22 sections, 3 equations, 5 figures, 11 tables)

This paper contains 22 sections, 3 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Summarization in Multi-Lingual World.
  4. Large Language Models.
  5. Data
  6. Experimental Setup
  7. Evaluation LLMs
  8. Evaluation Metrics
  9. Implementation Details
  10. Results and Analyses
  11. Discussion
  12. Conclusion
  13. Details of Intrinsic Metrics
  14. Data Contamination
  15. Data Statistics
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Illustration of many-to-many summarization.
  • Figure 2: Language-wise performance of tuned LLMs.
  • Figure 3: Length distributions of M2MS samples w.r.t different domains.
  • Figure 4: Illustration of the used M2MS prompt that includes a system round and a user round.
  • Figure 5: Model performance (ROUGE-1) using different scales of training samples.