Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends

Mina Taraghi, Gianolli Dorcelus, Armstrong Foundjem, Florian Tambon, Foutse Khomh

TL;DR

The paper investigates how practitioners reuse Pre-Trained Models within HuggingFace by combining a qualitative analysis of HF Forums with a quantitative examination of HF Hub trends. It identifies a 17-category taxonomy of challenges and a 6-category set of benefits, highlighting that model usage understanding, training pipelines, and output interpretability are dominant pain points, while expert guidance and collaboration provide key benefits. The study reveals that BERT-family models dominate both discussion and hub uploads, and that model-card documentation has not increased correspondingly despite tooling support, indicating persistence of comprehension and selection difficulties. By correlating forum mentions with hub uploads (notably a strong positive Spearman correlation around 0.70 for major models) and analyzing model-card adoption over time, the paper illuminates misalignments between community needs and platform tooling. It offers guidelines for platform providers and model contributors to improve onboarding, documentation, and community engagement, with implications extendable to other PTM hubs and software-reuse ecosystems.

Abstract

The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs. Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking. To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub. Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community. We then conduct a quantitative study to track model-type trends and model documentation evolution over time. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding. We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them. Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users. Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.

Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends

TL;DR

The paper investigates how practitioners reuse Pre-Trained Models within HuggingFace by combining a qualitative analysis of HF Forums with a quantitative examination of HF Hub trends. It identifies a 17-category taxonomy of challenges and a 6-category set of benefits, highlighting that model usage understanding, training pipelines, and output interpretability are dominant pain points, while expert guidance and collaboration provide key benefits. The study reveals that BERT-family models dominate both discussion and hub uploads, and that model-card documentation has not increased correspondingly despite tooling support, indicating persistence of comprehension and selection difficulties. By correlating forum mentions with hub uploads (notably a strong positive Spearman correlation around 0.70 for major models) and analyzing model-card adoption over time, the paper illuminates misalignments between community needs and platform tooling. It offers guidelines for platform providers and model contributors to improve onboarding, documentation, and community engagement, with implications extendable to other PTM hubs and software-reuse ecosystems.

Abstract

The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs, and dedicated platforms for hosting PTMs. Despite this trend, a comprehensive exploration of the challenges that users encounter and how the community leverages PTMs remains lacking. To address this gap, we conducted an extensive mixed-methods empirical study by focusing on discussion forums and the model hub of HuggingFace, the largest public model hub. Based on our qualitative analysis, we present a taxonomy of the challenges and benefits associated with PTM reuse within this community. We then conduct a quantitative study to track model-type trends and model documentation evolution over time. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding. We also identified interesting trends among models where some models maintain high upload rates despite a decline in topics related to them. Additionally, we found that despite the introduction of model documentation tools, its quantity has not increased over time, leading to difficulties in model comprehension and selection among users. Our study sheds light on new challenges in reusing PTMs that were not reported before and we provide recommendations for various stakeholders involved in PTM reuse.
Paper Structure (32 sections, 4 figures, 3 tables)

This paper contains 32 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Qualitative Analysis
  4. Motivation
  5. Methodology
  6. Data Collection
  7. Qualitative Analysis
  8. RQ1. What Challenges are Associated with Reusing PTMs in the HF Community?
  9. Taxonomy of Challenges
  10. Root Cause Analysis
  11. Comparison of Challenges based on the Expertise Level
  12. Analyzing the Trend of Challenges Over Time
  13. RQ2. What are the Benefits of the HF Community for Reusing PTMs?
  14. Results
  15. Quantitative Analysis
  16. ...and 17 more sections

Figures (4)

  • Figure 1: The mixed-methods methodology followed in the study.
  • Figure 2: The comparison of the percentage of some subcategories of challenges with "Beginner" and "Intermediate" tags
  • Figure 3: The ratio of the top-31 models in the forums and the model hub
  • Figure 4: The total number of models and the number of models with model cards uploaded on the hub over time (left axis). The ratio of the models with model cards over time (based on the model creation date) (right axis)