Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Long Text Understanding with Knowledge Distilled from Summarization Model

Yan Liu, Yazheng Yang, Xiaokang Chen

TL;DR

This work introduces Gist Detector, a compact encoder that learns to reproduce the gist signals of a powerful summarization model through knowledge distillation, producing a single, salient token-weight representation for long texts. By fusing this gist signal into downstream models via a simple fuse module, the approach enhances long-text understanding across document classification, distantly supervised open-domain QA, and non-parallel text style transfer. The method leverages an ensemble of summarization teachers to generate soft attention targets, and demonstrates consistent, significant improvements over strong baselines, supported by ablations. The resulting framework offers a practical path to incorporate abstractive summarization insights into diverse long-text tasks with reduced computation and model mismatch, indicating strong potential for broader applicability in NLP systems.

Abstract

Long text understanding is important yet challenging for natural language processing. A long article or document usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. With recent advances of abstractive summarization, we propose our \emph{Gist Detector} to leverage the gist detection ability of a summarization model and integrate the extracted gist into downstream models to enhance their long text understanding ability. Specifically, Gist Detector first learns the gist detection knowledge distilled from a summarization model, and then produces gist-aware representations to augment downstream models. We evaluate our method on three different tasks: long document classification, distantly supervised open-domain question answering, and non-parallel text style transfer. The experimental results show that our method can significantly improve the performance of baseline models on all tasks.

Improving Long Text Understanding with Knowledge Distilled from Summarization Model

TL;DR

This work introduces Gist Detector, a compact encoder that learns to reproduce the gist signals of a powerful summarization model through knowledge distillation, producing a single, salient token-weight representation for long texts. By fusing this gist signal into downstream models via a simple fuse module, the approach enhances long-text understanding across document classification, distantly supervised open-domain QA, and non-parallel text style transfer. The method leverages an ensemble of summarization teachers to generate soft attention targets, and demonstrates consistent, significant improvements over strong baselines, supported by ablations. The resulting framework offers a practical path to incorporate abstractive summarization insights into diverse long-text tasks with reduced computation and model mismatch, indicating strong potential for broader applicability in NLP systems.

Abstract

Long text understanding is important yet challenging for natural language processing. A long article or document usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. With recent advances of abstractive summarization, we propose our \emph{Gist Detector} to leverage the gist detection ability of a summarization model and integrate the extracted gist into downstream models to enhance their long text understanding ability. Specifically, Gist Detector first learns the gist detection knowledge distilled from a summarization model, and then produces gist-aware representations to augment downstream models. We evaluate our method on three different tasks: long document classification, distantly supervised open-domain question answering, and non-parallel text style transfer. The experimental results show that our method can significantly improve the performance of baseline models on all tasks.
Paper Structure (16 sections, 4 equations, 2 figures, 5 tables)

This paper contains 16 sections, 4 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Gist Detector Architecture
  4. Training with knowledge distillation
  5. Integration of salient information
  6. Experiments
  7. Distillation
  8. Integration into Downstream Tasks
  9. Document Classification:
  10. Distantly Supervised Open-Domain QA:
  11. Text Style Transfer:
  12. Results and Analysis
  13. Results on Document Classification
  14. Results on DS-QA
  15. Results on Text Style Transfer
  16. ...and 1 more sections

Figures (2)

  • Figure 1: An example from the CNN/Daily Mail dataset. The shading intensity represents the importance weight extracted from a well-trained summarization model.
  • Figure 2: Gist Detector is trained to reproduce the salient information from the teacher model, and provides the salient-aware representations as supplementary to augment the downstream model.