DRAGON: Robust Classification for Very Large Collections of Software Repositories

TL;DR

DRAGON tackles the challenge of robustly classifying very large software repository collections when rich metadata is sparse or missing. It uses a two-input, sentence-pair BERT model that separately processes file-tree signals and optional README text, trained with focal loss to handle long-tail domain topics drawn from the GitRanking taxonomy. The approach achieves state-of-the-art F1@5 on domain topics (60.8%) and remains robust to README absence, degrading only about 6–9% relative without retraining, with near-miss errors offering practical utility for search and discovery. In addition to strong results, the authors release the largest open dataset for repository classification (825k repos, 239 topics) and provide replication materials to support future large-scale, language-agnostic research on software repository understanding.

Abstract

The ability to automatically classify source code repositories with ''topics'' that reflect their content and purpose is very useful, especially when navigating or searching through large software collections. However, existing approaches often rely heavily on README files and other metadata, which are frequently missing, limiting their applicability in real-world large-scale settings. We present DRAGON, a repository classifier designed for very large and diverse software collections. It operates entirely on lightweight signals commonly stored in version control systems: file and directory names, and optionally the README when available. In repository classification at scale, DRAGON improves F1@5 from 54.8% to 60.8%, surpassing the state of the art. DRAGON remains effective even when README files are absent, with performance degrading by only 6% w.r.t. when they are present. This robustness makes it practical for real-world settings where documentation is sparse or inconsistent. Furthermore, many of the remaining classification errors are near misses, where predicted labels are semantically close to the correct topics. This property increases the practical value of the predictions in real-world software collections, where suggesting a few related topics can still guide search and discovery. As a byproduct of developing DRAGON, we also release the largest open dataset to date for repository classification, consisting of 825 thousand repositories with associated ground-truth topics, sourced from the Software Heritage archive, providing a foundation for future large-scale and language-agnostic research on software repository understanding.

Figures (8)

• Figure 1: Overview of the data processing and filtering pipeline. The Sankey diagram shows the progressive reduction of repositories from the initial 8047951.0 GitHub projects archived by Software Heritage with at least 1 topic, to the final dataset of approximately 825.0 000 repositories. Each split corresponds to a major operation described in the following subsections: \ref{['topic-filtering']} Topic filtering, \ref{['repository-mining']} Repository mining and \ref{['subsec:data_preparation']} Data cleaning.
• Figure 2: Topic distribution vs. power law ($y \propto x^{-0.77}$).
• Figure 3: Distribution of README lengths (in words) for our dataset, excluding outliers. Most repositories have minimal documentation.
• Figure 4: Distribution of the number of labels per repository. Most repositories have only one or two associated labels, showing the sparsity of topic annotations.
• Figure 5: Distribution of README lengths (in words) in the LEGION dataset. Compared to our dataset (median 75), LEGION READMEs are substantially longer (median 463 words), indicating higher documentation quality.