Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Neural Topic Models with Wasserstein Knowledge Distillation

Suman Adhya, Debarshi Kumar Sanyal

TL;DR

The paper tackles the memory footprint of large neural topic models by compressing a contextualized VAE-based topic model through Wasserstein knowledge distillation. It introduces a KD framework from a powerful teacher (CombinedTM) to a smaller student (ZeroShotTM) that jointly minimizes a $W_2$ distance between latent distributions and cross-entropy on soft labels, with the final student loss mixing the standard VAE objective and the KD objective. Across two public datasets, the distilled model achieves superior topic coherence (NPMI and CV) while substantially reducing parameters, and in some settings even surpasses the teacher. The approach enables more efficient deployment of neural topic models on resource-constrained devices and opens avenues for cross-model distillation in topic modeling; code is publicly available.

Abstract

Topic modeling is a dominant method for exploring document collections on the web and in digital libraries. Recent approaches to topic modeling use pretrained contextualized language models and variational autoencoders. However, large neural topic models have a considerable memory footprint. In this paper, we propose a knowledge distillation framework to compress a contextualized topic model without loss in topic quality. In particular, the proposed distillation objective is to minimize the cross-entropy of the soft labels produced by the teacher and the student models, as well as to minimize the squared 2-Wasserstein distance between the latent distributions learned by the two models. Experiments on two publicly available datasets show that the student trained with knowledge distillation achieves topic coherence much higher than that of the original student model, and even surpasses the teacher while containing far fewer parameters than the teacher's. The distilled model also outperforms several other competitive topic models on topic coherence.

Improving Neural Topic Models with Wasserstein Knowledge Distillation

TL;DR

The paper tackles the memory footprint of large neural topic models by compressing a contextualized VAE-based topic model through Wasserstein knowledge distillation. It introduces a KD framework from a powerful teacher (CombinedTM) to a smaller student (ZeroShotTM) that jointly minimizes a distance between latent distributions and cross-entropy on soft labels, with the final student loss mixing the standard VAE objective and the KD objective. Across two public datasets, the distilled model achieves superior topic coherence (NPMI and CV) while substantially reducing parameters, and in some settings even surpasses the teacher. The approach enables more efficient deployment of neural topic models on resource-constrained devices and opens avenues for cross-model distillation in topic modeling; code is publicly available.

Abstract

Topic modeling is a dominant method for exploring document collections on the web and in digital libraries. Recent approaches to topic modeling use pretrained contextualized language models and variational autoencoders. However, large neural topic models have a considerable memory footprint. In this paper, we propose a knowledge distillation framework to compress a contextualized topic model without loss in topic quality. In particular, the proposed distillation objective is to minimize the cross-entropy of the soft labels produced by the teacher and the student models, as well as to minimize the squared 2-Wasserstein distance between the latent distributions learned by the two models. Experiments on two publicly available datasets show that the student trained with knowledge distillation achieves topic coherence much higher than that of the original student model, and even surpasses the teacher while containing far fewer parameters than the teacher's. The distilled model also outperforms several other competitive topic models on topic coherence.
Paper Structure (6 sections, 6 equations, 2 figures, 3 tables)

This paper contains 6 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background on Wasserstein Distance
  3. Proposed Framework for Knowledge Distillation
  4. Experimental Setup
  5. Results
  6. Conclusion

Figures (2)

  • Figure 1: Framework for knowledge distillation from CombinedTM to ZeroShotTM.
  • Figure 2: Coherence scores (NPMI and CV) for different topic models on two datasets: 20NG and M10. The X-axis is marked with the topic counts used for each dataset.