Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization

Junjie Song, Yiwen Liu, Dapeng Li, Yin Sun, Shukun Fu, Siqi Chen, Yuji Cao

TL;DR

This paper tackles the challenge of balancing multiple quality dimensions in text summarization by introducing hypervolume optimization (HVO) for multi‑objective reinforcement learning. Building on group relative policy optimization (GRPO), HVO uses a hypervolume based reward to drive the model toward the Pareto frontier, improving balance across coherence, consistency, fluency, and relevance. Experiments on CNN/DailyMail and BillSum show that HVO achieves superior hypervolume and overall UniEval scores, with a 7B LLM enhanced by HVO performing comparably to GPT‑4 while generating shorter summaries. The approach does not rely on supervised fine tuning and provides open source code, highlighting a practical, Pareto‑aware framework for multi‑objective text summarization.

Abstract

Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model's optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives. Experimental results on several representative summarization datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at https://github.com/ai4business-LiAuto/HVO.git

Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization

TL;DR

This paper tackles the challenge of balancing multiple quality dimensions in text summarization by introducing hypervolume optimization (HVO) for multi‑objective reinforcement learning. Building on group relative policy optimization (GRPO), HVO uses a hypervolume based reward to drive the model toward the Pareto frontier, improving balance across coherence, consistency, fluency, and relevance. Experiments on CNN/DailyMail and BillSum show that HVO achieves superior hypervolume and overall UniEval scores, with a 7B LLM enhanced by HVO performing comparably to GPT‑4 while generating shorter summaries. The approach does not rely on supervised fine tuning and provides open source code, highlighting a practical, Pareto‑aware framework for multi‑objective text summarization.

Abstract

Text summarization is a crucial task that requires the simultaneous optimization of multiple objectives, including consistency, coherence, relevance, and fluency, which presents considerable challenges. Although large language models (LLMs) have demonstrated remarkable performance, enhanced by reinforcement learning (RL), few studies have focused on optimizing the multi-objective problem of summarization through RL based on LLMs. In this paper, we introduce hypervolume optimization (HVO), a novel optimization strategy that dynamically adjusts the scores between groups during the reward process in RL by using the hypervolume method. This method guides the model's optimization to progressively approximate the pareto front, thereby generating balanced summaries across multiple objectives. Experimental results on several representative summarization datasets demonstrate that our method outperforms group relative policy optimization (GRPO) in overall scores and shows more balanced performance across different dimensions. Moreover, a 7B foundation model enhanced by HVO performs comparably to GPT-4 in the summarization task, while maintaining a shorter generation length. Our code is publicly available at https://github.com/ai4business-LiAuto/HVO.git
Paper Structure (10 sections, 5 equations, 4 figures, 1 table)

This paper contains 10 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. METHOD
  3. Problem Formulation
  4. HVO
  5. Experiment
  6. Experiment Setup
  7. Dataset and Baselines
  8. Setup and Metrics
  9. Results
  10. Conclusion

Figures (4)

  • Figure 1: The radar chart shows the scores of different models in each dimension, evaluated by UniEval DBLP:conf/emnlp/Zhong0YMJLZJH22 on the CNN/DailyMail nallapati-etal-2016-abstractive. The same meanings are represented in the same color and the highlighted areas show where the HVO summary performs better. Underline indicates improper phrasing.
  • Figure 2: The entire process of HVO. In subplot (c), the points on the $f_1$ line represent the same linear weighted sum score for $D_1$ and $D_2$, while the points on the $f_2$ line represent the same hypervolume value for $D_1$ and $D_2$.
  • Figure 3: The inter-group means and standard deviations on the 500-validation set during the training process on CNN/DailyMail, with HVO method recording scores prior to hypervolume calculation and being comparable to GRPO.
  • Figure 4: Scatter plot illustrating the relationship between the overall score and completion length, as evaluated by UniEval, for different models on two datasets: CNN/DailyMail and BillSum.