Automatic Bottom-Up Taxonomy Construction: A Software Application Domain Study

TL;DR

This study tackles the challenge of missing explicit IS-A hierarchies in software classifications by building a bottom-up taxonomy of software application domains. It combines multiple data sources—GitRanking seeds, CSO, Wikidata—and Large Language Models in a hybrid pipeline to construct, optimize, and ensemble taxonomies, evaluated through automatic metrics, human annotation, and LLM-based assessments. The results show distinct strengths and weaknesses for each datasource, with the cascade ensemble plus LLM completion delivering a balanced, connected taxonomy that outperforms any single source. The work demonstrates the practical value of ensemble taxonomy construction for software classification and points to future integrations with AutoFL and broader applicability to tooling and code understanding tasks.

Abstract

Previous research in software application domain classification has faced challenges due to the lack of a proper taxonomy that explicitly models relations between classes. As a result, current solutions are less effective for real-world usage. This study aims to develop a comprehensive software application domain taxonomy by integrating multiple datasources and leveraging ensemble methods. The goal is to overcome the limitations of individual sources and configurations by creating a more robust, accurate, and reproducible taxonomy. This study employs a quantitative research design involving three different datasources: an existing Computer Science Ontology (CSO), Wikidata, and LLMs. The study utilises a combination of automated and human evaluations to assess the quality of a taxonomy. The outcome measures include the number of unlinked terms, self-loops, and overall connectivity of the taxonomy. The results indicate that individual datasources have advantages and drawbacks: the CSO datasource showed minimal variance across different configurations, but a notable issue of missing technical terms and a high number of self-loops. The Wikipedia datasource required significant filtering during construction to improve metric performance. LLM-generated taxonomies demonstrated better performance when using context-rich prompts. An ensemble approach showed the most promise, successfully reducing the number of unlinked terms and self-loops, thus creating a more connected and comprehensive taxonomy. The study addresses the construction of a software application domain taxonomy relying on pre-existing resources. Our results indicate that an ensemble approach to taxonomy construction can effectively address the limitations of individual datasources. Future work should focus on refining the ensemble techniques and exploring additional datasources to enhance the taxonomy's accuracy and completeness.

Figures (6)

• Figure 1: Pipeline: i) select terms, ii) optimise relationships per datasource, iii) combine results, and iv) evaluate.
• Figure 2: Percentage of correctly linked terms to Wikidata in CSO at various similarity thresholds.
• Figure 3: Results for each datasource include a distribution based on various hyperparameter configurations.
• Figure 4: Intersection between models for different metrics.
• Figure 5: Results for the ensemble models union (Un), cascade (Csc) and the final taxonomy (Fin).