Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization

Léo Hemamou, Mehdi Debiane

TL;DR

The paper addresses long-text extractive summarization by leveraging large language models (LLMs) beyond standard encoder-only architectures. It introduces EYEGLAXS, which fine-tunes LLAMA2-7B-32K-Instruct and ChatGLM2-6B-32K using LoRA, together with Flash Attention 2 and Rotary Positional Encoding to handle extremely long documents while performing sentence-level extraction via mean pooling and a lightweight classifier. The approach achieves state-of-the-art ROUGE scores on PubMed and arXiv among extractive methods and includes thorough analyses of sequence-length adaptability and small-data training scenarios, along with position-bias considerations. The work demonstrates the viability and practicality of using LLMs for long-text extractive summarization and points to future enhancements such as sliding attention and graph-based or reinforcement-learning integrations to further improve robustness and applicability.

Abstract

In an era where digital text is proliferating at an unprecedented rate, efficient summarization tools are becoming indispensable. While Large Language Models (LLMs) have been successfully applied in various NLP tasks, their role in extractive text summarization remains underexplored. This paper introduces EYEGLAXS (Easy Yet Efficient larGe LAnguage model for eXtractive Summarization), a framework that leverages LLMs, specifically LLAMA2-7B and ChatGLM2-6B, for extractive summarization of lengthy text documents. Instead of abstractive methods, which often suffer from issues like factual inaccuracies and hallucinations, EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. Utilizing state-of-the-art techniques such as Flash Attention and Parameter-Efficient Fine-Tuning (PEFT), EYEGLAXS addresses the computational and resource challenges typically associated with LLMs. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv. Furthermore, we extend our research through additional analyses that explore the adaptability of LLMs in handling different sequence lengths and their efficiency in training on smaller datasets. These contributions not only set a new standard in the field but also open up promising avenues for future research in extractive text summarization.

Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization

TL;DR

The paper addresses long-text extractive summarization by leveraging large language models (LLMs) beyond standard encoder-only architectures. It introduces EYEGLAXS, which fine-tunes LLAMA2-7B-32K-Instruct and ChatGLM2-6B-32K using LoRA, together with Flash Attention 2 and Rotary Positional Encoding to handle extremely long documents while performing sentence-level extraction via mean pooling and a lightweight classifier. The approach achieves state-of-the-art ROUGE scores on PubMed and arXiv among extractive methods and includes thorough analyses of sequence-length adaptability and small-data training scenarios, along with position-bias considerations. The work demonstrates the viability and practicality of using LLMs for long-text extractive summarization and points to future enhancements such as sliding attention and graph-based or reinforcement-learning integrations to further improve robustness and applicability.

Abstract

In an era where digital text is proliferating at an unprecedented rate, efficient summarization tools are becoming indispensable. While Large Language Models (LLMs) have been successfully applied in various NLP tasks, their role in extractive text summarization remains underexplored. This paper introduces EYEGLAXS (Easy Yet Efficient larGe LAnguage model for eXtractive Summarization), a framework that leverages LLMs, specifically LLAMA2-7B and ChatGLM2-6B, for extractive summarization of lengthy text documents. Instead of abstractive methods, which often suffer from issues like factual inaccuracies and hallucinations, EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. Utilizing state-of-the-art techniques such as Flash Attention and Parameter-Efficient Fine-Tuning (PEFT), EYEGLAXS addresses the computational and resource challenges typically associated with LLMs. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv. Furthermore, we extend our research through additional analyses that explore the adaptability of LLMs in handling different sequence lengths and their efficiency in training on smaller datasets. These contributions not only set a new standard in the field but also open up promising avenues for future research in extractive text summarization.
Paper Structure (26 sections, 8 equations, 3 figures, 6 tables)

This paper contains 26 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Litterature Review
  3. Long Extractive Text Summarization
  4. Large Language Models
  5. Parameter-Efficient Fine-Tuning methods
  6. Method
  7. Task Definition
  8. Choice of Large Language Models
  9. Transformer
  10. Query Key Value Projection Layers with LoRA
  11. Rotary Positional Encoding
  12. Self-Attention Module and Output Projection
  13. Mean Pooling and Classification Layer
  14. Experiments
  15. Datasets
  16. ...and 11 more sections

Figures (3)

  • Figure 1: The overall framework of EYEGLAXS. Residual connections and normalizations do not appear for better readability. Snowflake logo means that weights are frozen, while Fire logo means that weights are trainable.
  • Figure 2: Number of sentences selected at each relative position by the EYEGLAXS models and baselines compared to the oracles
  • Figure 3: ROUGE-2 F1 Measure scores for Longformer (Lodoss-base), ChatGLM2-LoRA (4K) and LLAMA2-LoRA across varying training data sizes. The exact number of training instances is indicated in parentheses.