Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SumRank: Aligning Summarization Models for Long-Document Listwise Reranking

Jincheng Feng, Wenhan Liu, Zhicheng Dou

Abstract

Large Language Models (LLMs) have demonstrated superior performance in listwise passage reranking task. However, directly applying them to rank long-form documents introduces both effectiveness and efficiency issues due to the substantially increased context length. To address this challenge, we propose a pointwise summarization model SumRank, aligned with downstream listwise reranking, to compress long-form documents into concise rank-aligned summaries before the final listwise reranking stage. To obtain our summarization model SumRank, we introduce a three-stage training pipeline comprising cold-start Supervised Fine-Tuning (SFT), specialized RL data construction, and rank-driven alignment via Reinforcement Learning. This paradigm aligns the SumRank with downstream ranking objectives to preserve relevance signals. We conduct extensive experiments on five benchmark datasets from the TREC Deep Learning tracks (TREC DL 19-23). Results show that our lightweight SumRank model achieves state-of-the-art (SOTA) ranking performance while significantly improving efficiency by reducing both summarization overhead and reranking complexity.

SumRank: Aligning Summarization Models for Long-Document Listwise Reranking

Abstract

Large Language Models (LLMs) have demonstrated superior performance in listwise passage reranking task. However, directly applying them to rank long-form documents introduces both effectiveness and efficiency issues due to the substantially increased context length. To address this challenge, we propose a pointwise summarization model SumRank, aligned with downstream listwise reranking, to compress long-form documents into concise rank-aligned summaries before the final listwise reranking stage. To obtain our summarization model SumRank, we introduce a three-stage training pipeline comprising cold-start Supervised Fine-Tuning (SFT), specialized RL data construction, and rank-driven alignment via Reinforcement Learning. This paradigm aligns the SumRank with downstream ranking objectives to preserve relevance signals. We conduct extensive experiments on five benchmark datasets from the TREC Deep Learning tracks (TREC DL 19-23). Results show that our lightweight SumRank model achieves state-of-the-art (SOTA) ranking performance while significantly improving efficiency by reducing both summarization overhead and reranking complexity.
Paper Structure (31 sections, 6 equations, 3 figures, 3 tables)

This paper contains 31 sections, 6 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Document Reranking with LLM
  4. Long Document Modeling
  5. Methodology
  6. Problem Formulation
  7. Cold-Start SFT
  8. RL Data Construction
  9. Rank-Driven Alignment via GRPO
  10. Target Selection and Rollout
  11. Listwise Evaluation and Reward
  12. Optimization Objective
  13. Experimental Setup
  14. Datasets and Metrics
  15. Baselines
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The left part shows the method of naive LLM-based long document listwise rerank. The right part shows our proposed SumRank, which is better than the naive listwise method both in effectiveness and efficiency. $\text{Sum}_1, \text{Sum}_2, \dots, \text{Sum}_N$ denote the summaries generated by SumRank.
  • Figure 2: The overall architecture of SumRank, which consists of three training stages and the Summary-then-Rank pipeline (Listwise Rerank with Pointwise Summary).
  • Figure 3: Ranking latency (seconds/query) across different methods on all five TREC DL benchmarks (DL19–23).