SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization

TL;DR

This paper introduces SummaReranker, a second-stage, mixture-of-experts re-ranking framework that selects the best candidate from a pool produced by a base abstractive summarization model and diverse decoding methods. By framing candidate selection as per-metric binary decisions and sharing experts across multiple evaluation metrics, the model achieves robust, multi-task optimization while remaining efficient. Across CNN/DM, XSum, and Reddit TIFU, SummaReranker yields state-of-the-art ROUGE results and gains in human-judged fidelity, while also increasing abstractiveness of the outputs. The approach demonstrates strong transferability in the transfer setup and provides practical guidance on decoding-method selection and inference efficiency, highlighting its potential for real-world summarization systems.

Abstract

Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially through fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search to generate a unique summary. However, the search space is very large, and with the exposure bias, such decoding is not optimal. In this paper, we show that it is possible to directly train a second-stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candidate and consistently improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art. Our code and checkpoints will be available at https://github.com/ntunlp/SummaReranker.

Figures (9)

• Figure 1: SummaReranker model architecture, optimizing $N$ metrics. The summarization metrics here (ROUGE-1, ROUGE-2, ..., BARTScore) are displayed as examples.
• Figure 2: Expert utilization for a base PEGASUS with SummaReranker optimized with {R-1, R-2, R-L, BS, BaS} on CNN/DM, with 10 experts.
• Figure 3: Best summary candidate recall with 15 diverse beam search candidates for PEGASUS on all three datasets. SR denotes SummaReranker. Dotted lines are random baselines, and dashed lines correspond to the base PEGASUS.
• Figure 4: Example of a summary generated by SummaReranker trained for {R-1, R-2, R-L} on CNN/DM. The sentence in green is included in the SummaReranker summary, while the one in red is discarded.
• Figure 5: Human evaluation results on all three datasets. Black vertical bars are standard deviation across human raters.