Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Key-Element-Informed sLLM Tuning for Document Summarization

Sangwon Ryu, Heejin Do, Yunsu Kim, Gary Geunbae Lee, Jungseul Ok

TL;DR

A key-element-informed instruction tuning for summarization, so-called KEITSum, is proposed, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements, competitive to proprietary LLM.

Abstract

Remarkable advances in large language models (LLMs) have enabled high-quality text summarization. However, this capability is currently accessible only through LLMs of substantial size or proprietary LLMs with usage fees. In response, smaller-scale LLMs (sLLMs) of easy accessibility and low costs have been extensively studied, yet they often suffer from missing key information and entities, i.e., low relevance, in particular, when input documents are long. We hence propose a key-element-informed instruction tuning for summarization, so-called KEITSum, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements. Experimental results on dialogue and news datasets demonstrate that sLLM with KEITSum indeed provides high-quality summarization with higher relevance and less hallucinations, competitive to proprietary LLM.

Key-Element-Informed sLLM Tuning for Document Summarization

TL;DR

A key-element-informed instruction tuning for summarization, so-called KEITSum, is proposed, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements, competitive to proprietary LLM.

Abstract

Remarkable advances in large language models (LLMs) have enabled high-quality text summarization. However, this capability is currently accessible only through LLMs of substantial size or proprietary LLMs with usage fees. In response, smaller-scale LLMs (sLLMs) of easy accessibility and low costs have been extensively studied, yet they often suffer from missing key information and entities, i.e., low relevance, in particular, when input documents are long. We hence propose a key-element-informed instruction tuning for summarization, so-called KEITSum, which identifies key elements in documents and instructs sLLM to generate summaries capturing these key elements. Experimental results on dialogue and news datasets demonstrate that sLLM with KEITSum indeed provides high-quality summarization with higher relevance and less hallucinations, competitive to proprietary LLM.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Key-element-informed tuning
  4. Key-element extraction
  5. Instruction tuning
  6. Experimental setup
  7. Results and discussions
  8. Main results
  9. Length dependency
  10. Human evaluation
  11. Measuring hallucinations
  12. Conclusion
  13. Acknowledgements

Figures (3)

  • Figure 1: Description of KEITSum Framework. We extract named entities and conclusion sentence from the source document and insert emphasis tokens. Following this, we create a full description by adding detailed instructions.
  • Figure 2: The proportion of entities included in the element-aware dataset that are also included in the summaries generated by each model.
  • Figure 3: Hallucination ratio per dialog in DialogSum.