Towards Leveraging Large Language Model Summaries for Topic Modeling in Source Code
Michele Carissimi, Martina Saletta, Claudio Ferretti
TL;DR
This paper tackles topic modeling for source code in the absence of natural-language cues by proposing a workflow that uses an LLM to generate summaries of Python functions whose names are obfuscated and whose comments are removed, followed by BERTopic-based topic modeling on these summaries. It introduces four robustness distances ($d_{MSE}$, $d_{TOP}$, $d_{TOPw}$, $d_\cap$) to compare topics inferred from summaries with those derived from docstrings or function names. Experiments on $N=10{,}000$ Python functions from CodeSearchNet show that topics learned from LLM-generated summaries achieve a moderate-to-high interpretability ($C_v \approx 0.60$) and align closely with reference topics from documentation, outperforming approaches based on obfuscated identifiers. The approach demonstrates robustness to missing textual cues and offers practical benefits for automatic documentation, tagging, code search, and large-repository knowledge discovery. The study also provides a framework for evaluating topic-consistency across representations and suggests avenues for extending the method to additional languages and tasks.
Abstract
Understanding source code is a topic of great interest in the software engineering community, since it can help programmers in various tasks such as software maintenance and reuse. Recent advances in large language models (LLMs) have demonstrated remarkable program comprehension capabilities, while transformer-based topic modeling techniques offer effective ways to extract semantic information from text. This paper proposes and explores a novel approach that combines these strengths to automatically identify meaningful topics in a corpus of Python programs. Our method consists in applying topic modeling on the descriptions obtained by asking an LLM to summarize the code. To assess the internal consistency of the extracted topics, we compare them against topics inferred from function names alone, and those derived from existing docstrings. Experimental results suggest that leveraging LLM-generated summaries provides interpretable and semantically rich representation of code structure. The promising results suggest that our approach can be fruitfully applied in various software engineering tasks such as automatic documentation and tagging, code search, software reorganization and knowledge discovery in large repositories.