Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling

Yida Mu, Chun Dong, Kalina Bontcheva, Xingyi Song

TL;DR

Traditional topic modelling struggles with semantic understanding and overlapping topics, particularly for unseen topics. This work proposes a prompt-driven framework where large language models (LLMs) generate topics directly from documents, paired with evaluation protocols to assess clustering quality. Through experiments with GPT-3.5 and LLaMA-2-7B on open-domain (20NG) and domain-specific (CAVS) data, the authors demonstrate that LLMs can produce interpretable, granular topics and even summarise them into final topic lists, aided by seed-guidance and summarisation steps. The authors also introduce novel evaluation metrics (topic distance, semantic similarity, recall/precision) and present a temporal case study on COVID-19 vaccine hesitancy to show practical applicability for dynamic corpora.

Abstract

Topic modelling, as a well-established unsupervised technique, has found extensive use in automatically detecting significant topics within a corpus of documents. However, classic topic modelling approaches (e.g., LDA) have certain drawbacks, such as the lack of semantic understanding and the presence of overlapping topics. In this work, we investigate the untapped potential of large language models (LLMs) as an alternative for uncovering the underlying topics within extensive text corpora. To this end, we introduce a framework that prompts LLMs to generate topics from a given set of documents and establish evaluation protocols to assess the clustering efficacy of LLMs. Our findings indicate that LLMs with appropriate prompts can stand out as a viable alternative, capable of generating relevant topic titles and adhering to human guidelines to refine and merge topics. Through in-depth experiments and evaluation, we summarise the advantages and constraints of employing LLMs in topic extraction.

Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling

TL;DR

Traditional topic modelling struggles with semantic understanding and overlapping topics, particularly for unseen topics. This work proposes a prompt-driven framework where large language models (LLMs) generate topics directly from documents, paired with evaluation protocols to assess clustering quality. Through experiments with GPT-3.5 and LLaMA-2-7B on open-domain (20NG) and domain-specific (CAVS) data, the authors demonstrate that LLMs can produce interpretable, granular topics and even summarise them into final topic lists, aided by seed-guidance and summarisation steps. The authors also introduce novel evaluation metrics (topic distance, semantic similarity, recall/precision) and present a temporal case study on COVID-19 vaccine hesitancy to show practical applicability for dynamic corpora.

Abstract

Topic modelling, as a well-established unsupervised technique, has found extensive use in automatically detecting significant topics within a corpus of documents. However, classic topic modelling approaches (e.g., LDA) have certain drawbacks, such as the lack of semantic understanding and the presence of overlapping topics. In this work, we investigate the untapped potential of large language models (LLMs) as an alternative for uncovering the underlying topics within extensive text corpora. To this end, we introduce a framework that prompts LLMs to generate topics from a given set of documents and establish evaluation protocols to assess the clustering efficacy of LLMs. Our findings indicate that LLMs with appropriate prompts can stand out as a viable alternative, capable of generating relevant topic titles and adhering to human guidelines to refine and merge topics. Through in-depth experiments and evaluation, we summarise the advantages and constraints of employing LLMs in topic extraction.
Paper Structure (27 sections, 4 figures, 1 table)

This paper contains 27 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Topic Modelling
  4. Close-set Topic Classification
  5. LLMs-driven Topic Extraction
  6. Our Work
  7. Models and Datasets
  8. LLMs
  9. Datasets
  10. Experiments
  11. Experiment 1: Out-of-box (Basic Prompt)
  12. Prompting Strategies
  13. Results and Discussion
  14. Solutions
  15. Experiment 2: Topics Granularity (GPT & LLaMA Expt. 2 + Seeds Topic)
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Example prompts for LLaMA. Text enclosed by the special tokens '$\ll$SYS$\gg$' is designated as a system prompt.
  • Figure 2: Two examples of final topics summarised by LLMs, namely 'Technology & Computers' (20NG) and 'Trust & Mistrust' (Vaccine dataset).
  • Figure 3: Matrix showing the Jaccard Distance between sub-topics derived from the top 10 general topic in the final list.
  • Figure 4: The figure showcases a trio of word clouds (upper half), each capturing the predominant reasons for vaccine hesitancy related to the COVID-19 vaccine over distinct time phases. Additionally, using the cutting-edge multi-modal GPT-4, we obtain descriptions (bottom half) derived from these word clouds.