Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation

Yichao Feng, Shuai Zhao, Yueqiu Li, Luwei Xiao, Xiaobao Wu, Anh Tuan Luu

TL;DR

This paper tackles aspect-based summarization by addressing limitations of standard LLM-based methods, such as restricted input length and prompt-driven token usage that fuel hallucinations. It introduces Self-Aspect Retrieval Enhanced Summary Generation (SARESG), a dense-embedding retrieval and pruning framework that extracts content relevant to a given aspect, prunes unrelated text to respect token limits, and uses reranking to sharpen context before generation. Across USB, OAsum, and Ma-news datasets, SARESG consistently improves aspect-aligned summarization metrics (METEOR, ROUGE, BERTScore) and enables effective use of in-context learning via preserved token space. The work highlights the benefits of chunk-level retrieval, analyzes chunk-size effects, and demonstrates that retrieval-guided ICL can yield robust performance across model sizes, while noting limitations in ICL stability and resource demands that warrant future optimization.

Abstract

Aspect-based summarization aims to generate summaries tailored to specific aspects, addressing the resource constraints and limited generalizability of traditional summarization approaches. Recently, large language models have shown promise in this task without the need for training. However, they rely excessively on prompt engineering and face token limits and hallucination challenges, especially with in-context learning. To address these challenges, in this paper, we propose a novel framework for aspect-based summarization: Self-Aspect Retrieval Enhanced Summary Generation. Rather than relying solely on in-context learning, given an aspect, we employ an embedding-driven retrieval mechanism to identify its relevant text segments. This approach extracts the pertinent content while avoiding unnecessary details, thereby mitigating the challenge of token limits. Moreover, our framework optimizes token usage by deleting unrelated parts of the text and ensuring that the model generates output strictly based on the given aspect. With extensive experiments on benchmark datasets, we demonstrate that our framework not only achieves superior performance but also effectively mitigates the token limitation problem.

Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation

TL;DR

This paper tackles aspect-based summarization by addressing limitations of standard LLM-based methods, such as restricted input length and prompt-driven token usage that fuel hallucinations. It introduces Self-Aspect Retrieval Enhanced Summary Generation (SARESG), a dense-embedding retrieval and pruning framework that extracts content relevant to a given aspect, prunes unrelated text to respect token limits, and uses reranking to sharpen context before generation. Across USB, OAsum, and Ma-news datasets, SARESG consistently improves aspect-aligned summarization metrics (METEOR, ROUGE, BERTScore) and enables effective use of in-context learning via preserved token space. The work highlights the benefits of chunk-level retrieval, analyzes chunk-size effects, and demonstrates that retrieval-guided ICL can yield robust performance across model sizes, while noting limitations in ICL stability and resource demands that warrant future optimization.

Abstract

Aspect-based summarization aims to generate summaries tailored to specific aspects, addressing the resource constraints and limited generalizability of traditional summarization approaches. Recently, large language models have shown promise in this task without the need for training. However, they rely excessively on prompt engineering and face token limits and hallucination challenges, especially with in-context learning. To address these challenges, in this paper, we propose a novel framework for aspect-based summarization: Self-Aspect Retrieval Enhanced Summary Generation. Rather than relying solely on in-context learning, given an aspect, we employ an embedding-driven retrieval mechanism to identify its relevant text segments. This approach extracts the pertinent content while avoiding unnecessary details, thereby mitigating the challenge of token limits. Moreover, our framework optimizes token usage by deleting unrelated parts of the text and ensuring that the model generates output strictly based on the given aspect. With extensive experiments on benchmark datasets, we demonstrate that our framework not only achieves superior performance but also effectively mitigates the token limitation problem.

Paper Structure

This paper contains 16 sections, 4 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aspect-based Summarization
  4. LLM-based Summarization
  5. Methodology
  6. Sentences retrieval
  7. Experiments
  8. Experiment Setup
  9. Zero-shot Generation
  10. One-shot Generation with Selective Context
  11. Results
  12. Main Results
  13. Influence of sentence retrieval
  14. Influence of chunk size
  15. ICL sample
  16. ...and 1 more sections

Figures (3)

  • Figure 1: This diagram illustrates the generation process of the system. The model first evaluates the overall length of the input text. If the text is shorter than a predefined threshold, it is directly fed into the LLM for summary generation. For longer texts, the system splits the input into smaller chunks, uses an embedding model to assign relevance scores to each chunk, filters out the least relevant parts, and reconstructs the remaining content in its original order to ensure readability and coherence for the model.
  • Figure 2: Experiment results for three different chunk sizes on MA-news dataset. Results generated with Llama3-70b.
  • Figure 3: Radar charts were created for all ICL samples tested on the MA-news dataset, with results generated using the Llama3-8b model. Blue is representing R1, orange is representing R2, and green is representing Rl. The captions represent the aspects of the samples, with six points on each chart corresponding to the aspects of the testing set.