Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models

TL;DR

This paper tackles the challenge of evaluating evolving topic models by introducing a purpose-oriented framework that leverages multiple open-source LLMs to assess topic quality along four dimensions. It defines nine LLM-based metrics mapped to three real-world purposes: content understanding, document labeling, and retrieval/summarization, and validates them through adversarial and sampling protocols across three diverse datasets and four topic-modeling approaches. The findings show that LLM-based metrics provide interpretable, robust, and task-relevant diagnostics, revealing weaknesses such as redundancy and semantic drift that traditional coherence/diversity scores miss. The work demonstrates the complementarity of LLM judgments with conventional metrics and offers practical guidance for model selection, diagnosis, and quality assurance in dynamic knowledge environments, with code and data openly available at the provided GitHub repository.

Abstract

This study presents a framework for automated evaluation of dynamically evolving topic models using Large Language Models (LLMs). Topic modeling is essential for organizing and retrieving scholarly content in digital library systems, helping users navigate complex and evolving knowledge domains. However, widely used automated metrics, such as coherence and diversity, often capture only narrow statistical patterns and fail to explain semantic failures in practice. We introduce a purpose-oriented evaluation framework that employs nine LLM-based metrics spanning four key dimensions of topic quality: lexical validity, intra-topic semantic soundness, inter-topic structural soundness, and document-topic alignment soundness. The framework is validated through adversarial and sampling-based protocols, and is applied across datasets spanning news articles, scholarly publications, and social media posts, as well as multiple topic modeling methods and open-source LLMs. Our analysis shows that LLM-based metrics provide interpretable, robust, and task-relevant assessments, uncovering critical weaknesses in topic models such as redundancy and semantic drift, which are often missed by traditional metrics. These results support the development of scalable, fine-grained evaluation tools for maintaining topic relevance in dynamic datasets. All code and data supporting this work are accessible at https://github.com/zhiyintan/topic-model-LLMjudgment.

Figures (4)

• Figure 1: Overview of our LLM-based evaluation metrics. Group by different evaluation dimensions, there are rating-based and counting-based metrics.
• Figure 2: Result of LLM-based Metrics (rating).
• Figure 3: Result of LLM-based Metrics (counting).
• Figure 4: Correlation Heatmap of LLM-based Metrics