Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ProtoTopic: Prototypical Network for Few-Shot Medical Topic Modeling

Martin Licht, Sara Ketabi, Farzad Khalvati

TL;DR

The paper tackles the challenge of topic modeling in medical text under data scarcity by introducing ProtoTopic, a prototypical-network based framework that learns topic prototypes from few examples. It uses embeddings from PubMedBERT and all-MiniLM-L6-v2, clusters via K-means to create pseudo-labels, and trains a prototypical network that assigns topics by prototype distances, followed by c-TF-IDF keyword extraction. Experiments on PubMed200k show ProtoTopic achieving higher topic coherence and diversity than LDA and BERTopic across 25, 50, and 100 topics, with statistical significance (p < 1e-5) versus BERTopic. The work demonstrates the viability of few-shot prototypical networks for medical topic modeling and points to future gains from alternative clustering, improved loss terms, dimensionality reduction, and clinician-based evaluations.

Abstract

Topic modeling is a useful tool for analyzing large corpora of written documents, particularly academic papers. Despite a wide variety of proposed topic modeling techniques, these techniques do not perform well when applied to medical texts. This can be due to the low number of documents available for some topics in the healthcare domain. In this paper, we propose ProtoTopic, a prototypical network-based topic model used for topic generation for a set of medical paper abstracts. Prototypical networks are efficient, explainable models that make predictions by computing distances between input datapoints and a set of prototype representations, making them particularly effective in low-data or few-shot learning scenarios. With ProtoTopic, we demonstrate improved topic coherence and diversity compared to two topic modeling baselines used in the literature, demonstrating the ability of our model to generate medically relevant topics even with limited data.

ProtoTopic: Prototypical Network for Few-Shot Medical Topic Modeling

TL;DR

The paper tackles the challenge of topic modeling in medical text under data scarcity by introducing ProtoTopic, a prototypical-network based framework that learns topic prototypes from few examples. It uses embeddings from PubMedBERT and all-MiniLM-L6-v2, clusters via K-means to create pseudo-labels, and trains a prototypical network that assigns topics by prototype distances, followed by c-TF-IDF keyword extraction. Experiments on PubMed200k show ProtoTopic achieving higher topic coherence and diversity than LDA and BERTopic across 25, 50, and 100 topics, with statistical significance (p < 1e-5) versus BERTopic. The work demonstrates the viability of few-shot prototypical networks for medical topic modeling and points to future gains from alternative clustering, improved loss terms, dimensionality reduction, and clinician-based evaluations.

Abstract

Topic modeling is a useful tool for analyzing large corpora of written documents, particularly academic papers. Despite a wide variety of proposed topic modeling techniques, these techniques do not perform well when applied to medical texts. This can be due to the low number of documents available for some topics in the healthcare domain. In this paper, we propose ProtoTopic, a prototypical network-based topic model used for topic generation for a set of medical paper abstracts. Prototypical networks are efficient, explainable models that make predictions by computing distances between input datapoints and a set of prototype representations, making them particularly effective in low-data or few-shot learning scenarios. With ProtoTopic, we demonstrate improved topic coherence and diversity compared to two topic modeling baselines used in the literature, demonstrating the ability of our model to generate medically relevant topics even with limited data.
Paper Structure (14 sections, 2 equations, 5 figures, 3 tables)

This paper contains 14 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Topic Modeling
  4. Few-shot Learning
  5. Prototypical Networks
  6. Materials and Methods
  7. Data and Data Preprocessing
  8. Methodology
  9. Evaluation Setting
  10. Results
  11. Quantitative Results
  12. Qualitative Results
  13. Discussion
  14. Conclusion

Figures (5)

  • Figure 1: ProtoTopic Training Pipeline
  • Figure 2: A Schematic Overview of the Proposed ProtoTopic Model: Abstracts are shown in a 2D representation of the PubMedBERT/all-MiniLM-L6-v2 embedding space. Each of the 3 topics has 5 abstract embeddings in its support set. The prototypes (red, blue and grey points) are computed by taking the mean of the support set embeddings, and the query point (abstract in text box) is then compared to the prototypes to predict which topic it corresponds to.
  • Figure 3: Similarities between BERTopic (left) and ProtoTopic (right) keywords for "cancer" topic.
  • Figure 4: A comparison between the specificity of a set of keywords generated by BERTopic (left) and ProtoTopic (right).
  • Figure 5: A Schematic Overview of the Effect of Topic Number on Topic Diversity: Topics are shown in a 2D representation of the embedding space. We initially see 10 topics with low diversity (top left) due to general topics and 10 topics with high diversity (bottom left) and less topic overlap. As the topic number increases, the topic diversity converges.