Benchmarking Recommendation, Classification, and Tracing Based on Hugging Face Knowledge Graph

TL;DR

The paper addresses the fragmentation of open-source ML resource metadata by building HuggingKG, a large-scale heterogeneous knowledge graph derived from the Hugging Face ecosystem, containing $2.6$ million nodes and $6.2$ million edges. It pairs HuggingKG with HuggingBench, a multi-task benchmark for resource recommendation, task classification, and model tracing, leveraging both graph structure and rich textual attributes to improve discovery, organization, and lineage tracing of ML resources. Empirical results show that KG-based methods, text embeddings, and task-specific finetuning yield notable gains across the three IR tasks, while also revealing challenges in sparse, domain-specific graphs. The resources are publicly available to support reproducible research and foster advances in open-source resource management, model lineage tracing, and AI tool integration within large-scale ML ecosystems, with plans for ongoing updates and cross-platform extension.

Abstract

The rapid growth of open source machine learning (ML) resources, such as models and datasets, has accelerated IR research. However, existing platforms like Hugging Face do not explicitly utilize structured representations, limiting advanced queries and analyses such as tracing model evolution and recommending relevant datasets. To fill the gap, we construct HuggingKG, the first large-scale knowledge graph built from the Hugging Face community for ML resource management. With 2.6 million nodes and 6.2 million edges, HuggingKG captures domain-specific relations and rich textual attributes. It enables us to further present HuggingBench, a multi-task benchmark with three novel test collections for IR tasks including resource recommendation, classification, and tracing. Our experiments reveal unique characteristics of HuggingKG and the derived tasks. Both resources are publicly available, expected to advance research in open source resource sharing and management.

Figures (5)

• Figure 1: Illustration of $\mathsf{HuggingKG}$ and $\mathsf{HuggingBench}$.
• Figure 2: An example model page on Hugging Face.
• Figure 3: The schema graph of $\mathsf{HuggingKG}$, along with the quantity and proportion of each node and edge type.
• Figure 4: Conditional probability $P(A|B)$ of user co-likes.
• Figure 5: Description length of Model and Dataset.