Boosting Summarization with Normalizing Flows and Aggressive Training

TL;DR

FlowSUM advances Transformer-based summarization by embedding a flexible, normalizing flows–driven latent posterior within a variational encoder–decoder. The approach tackles latent information bottlenecks and posterior collapse with a CAAT training regime and a refined gating mechanism, achieving stronger ROUGE and semantic fidelity while enabling effective knowledge distillation with negligible inference cost. Empirical results across diverse datasets show gains on long-form summaries and robust performance when distilling knowledge from larger models, though short-sum tasks remain challenging. The work highlights the practical potential of flow-based latent modeling for text generation and suggests future extensions toward diffusion-based latent representations.

Abstract

This paper presents FlowSUM, a normalizing flows-based variational encoder-decoder framework for Transformer-based summarization. Our approach tackles two primary challenges in variational summarization: insufficient semantic information in latent representations and posterior collapse during training. To address these challenges, we employ normalizing flows to enable flexible latent posterior modeling, and we propose a controlled alternate aggressive training (CAAT) strategy with an improved gate mechanism. Experimental results show that FlowSUM significantly enhances the quality of generated summaries and unleashes the potential for knowledge distillation with minimal impact on inference time. Furthermore, we investigate the issue of posterior collapse in normalizing flows and analyze how the summary quality is affected by the training strategy, gate initialization, and the type and number of normalizing flows used, offering valuable insights for future research.

Figures (5)

• Figure 1: FlowSUM Model Architecture, including an NF latent module (in purple), a Transformer-based encoder-decoder (in green), and a refined gate mechanism (in orange)
• Figure 2: Comparison of training strategies and gate initialization.
• Figure 3: A closer look at the training process: CAAT vs. Standard Training.
• Figure 4: Visualization of the first two dimensions of $z_0$, $z_K$, and $N(0, I)$ by FlowSUM-PLKD on CNN/DM. The right sub-figure demonstrates a clear bi-modality.
• Figure 5: Visualization of the first two dimensions of $z_0$, $z_K$, and $N(0, I)$ by FlowSUM-PLKD on XSum. Both sub-figures demonstrate distinct multi-modal patterns.