Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lessons Learned from Mining the Hugging Face Repository

Joel Castaño, Silverio Martínez-Fernández, Xavier Franch

TL;DR

The paper addresses rigorous empirical study of the Hugging Face (HF) ecosystem by synthesizing lessons from two HF-focused investigations on carbon emissions and maintenance. It introduces a unified replication package, advocates a stratified sampling approach across domains, model sizes, and popularity, and proposes a cohort-study framework to approach causality in repository mining. Key contributions include practical data collection and preprocessing pipelines via the HFHub API, DistilBERT-based commit classification, and guidelines for longitudinal HF analyses. The work has practical impact by enabling transparent, scalable, and more generalizable empirical analyses of HF resources, supporting sustainable development and governance of large-scale language-model platforms.

Abstract

The rapidly evolving fields of Machine Learning (ML) and Artificial Intelligence have witnessed the emergence of platforms like Hugging Face (HF) as central hubs for model development and sharing. This experience report synthesizes insights from two comprehensive studies conducted on HF, focusing on carbon emissions and the evolutionary and maintenance aspects of ML models. Our objective is to provide a practical guide for future researchers embarking on mining software repository studies within the HF ecosystem to enhance the quality of these studies. We delve into the intricacies of the replication package used in our studies, highlighting the pivotal tools and methodologies that facilitated our analysis. Furthermore, we propose a nuanced stratified sampling strategy tailored for the diverse HF Hub dataset, ensuring a representative and comprehensive analytical approach. The report also introduces preliminary guidelines, transitioning from repository mining to cohort studies, to establish causality in repository mining studies, particularly within the ML model of HF context. This transition is inspired by existing frameworks and is adapted to suit the unique characteristics of the HF model ecosystem. Our report serves as a guiding framework for researchers, contributing to the responsible and sustainable advancement of ML, and fostering a deeper understanding of the broader implications of ML models.

Lessons Learned from Mining the Hugging Face Repository

TL;DR

The paper addresses rigorous empirical study of the Hugging Face (HF) ecosystem by synthesizing lessons from two HF-focused investigations on carbon emissions and maintenance. It introduces a unified replication package, advocates a stratified sampling approach across domains, model sizes, and popularity, and proposes a cohort-study framework to approach causality in repository mining. Key contributions include practical data collection and preprocessing pipelines via the HFHub API, DistilBERT-based commit classification, and guidelines for longitudinal HF analyses. The work has practical impact by enabling transparent, scalable, and more generalizable empirical analyses of HF resources, supporting sustainable development and governance of large-scale language-model platforms.

Abstract

The rapidly evolving fields of Machine Learning (ML) and Artificial Intelligence have witnessed the emergence of platforms like Hugging Face (HF) as central hubs for model development and sharing. This experience report synthesizes insights from two comprehensive studies conducted on HF, focusing on carbon emissions and the evolutionary and maintenance aspects of ML models. Our objective is to provide a practical guide for future researchers embarking on mining software repository studies within the HF ecosystem to enhance the quality of these studies. We delve into the intricacies of the replication package used in our studies, highlighting the pivotal tools and methodologies that facilitated our analysis. Furthermore, we propose a nuanced stratified sampling strategy tailored for the diverse HF Hub dataset, ensuring a representative and comprehensive analytical approach. The report also introduces preliminary guidelines, transitioning from repository mining to cohort studies, to establish causality in repository mining studies, particularly within the ML model of HF context. This transition is inspired by existing frameworks and is adapted to suit the unique characteristics of the HF model ecosystem. Our report serves as a guiding framework for researchers, contributing to the responsible and sustainable advancement of ML, and fostering a deeper understanding of the broader implications of ML models.
Paper Structure (25 sections, 2 figures)

This paper contains 25 sections, 2 figures.

Table of Contents

  1. Introduction
  2. BACKGROUND AND RELATED WORK
  3. Repository Mining
  4. Repository Mining in HF
  5. Sampling and Stratified Sampling
  6. Cohort Studies and Their Adaptation in Mining Software Repositories
  7. HF Mining: Methodological Insights
  8. Replication Package
  9. Data Collection
  10. Data Preprocessing
  11. Automation of the Data Update
  12. Considerations and Challenges
  13. Feasibility and Implementation
  14. Sampling
  15. Sampling Procedure
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Replication package diagram
  • Figure 2: Stratification sampling diagram