How to Train Text Summarization Model with Weak Supervisions

TL;DR

This work proposes a method that breaks down the complex objective into simpler tasks and generates supervision signals for each one, and integrates these supervision signals into a manageable form, resulting in a straightforward learning procedure.

Abstract

Currently, machine learning techniques have seen significant success across various applications. Most of these techniques rely on supervision from human-generated labels or a mixture of noisy and imprecise labels from multiple sources. However, for certain complex tasks, even noisy or inexact labels are unavailable due to the intricacy of the objectives. To tackle this issue, we propose a method that breaks down the complex objective into simpler tasks and generates supervision signals for each one. We then integrate these supervision signals into a manageable form, resulting in a straightforward learning procedure. As a case study, we demonstrate a system used for topic-based summarization. This system leverages rich supervision signals to promote both summarization and topic relevance. Remarkably, we can train the model end-to-end without any labels. Experimental results indicate that our approach performs exceptionally well on the CNN and DailyMail datasets.

Figures (2)

• Figure 1: The framework of this paper. The dashed line represents the learning procedure, and the solid line represents supervision generation. topic is extracted from the source document using NER. We don't modify neural architecture and regard it as a black box. Various supervisions are integrated as the learning target.
• Figure 2: weight and loss change over iterations. Note that weight are fixed after 7k iterations.