Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Model Hubs and Beyond: Analyzing Model Popularity, Performance, and Documentation

Pritam Kadasi, Sriman Reddy Kondam, Srivathsa Vamsi Chaturvedula, Rudranshu Sen, Agnish Saha, Soumavo Sikdar, Sayani Sarkar, Suhani Mittal, Rohit Jindal, Mayank Singh

TL;DR

The paper investigates whether popularity signals on Hugging Face align with real model performance and documentation quality by evaluating the top $M_{500}$ sentiment-analysis models across three benchmarks and a Reddit dataset. It combines large-scale manual inspection of model cards (approximately $78{,}000$ elements) with a dataset cartography framework to assess performance across Easy/Ambiguous/Hard instances, revealing weak correlations between popularity and performance and substantial documentation gaps. A case study on Reddit demonstrates poor generalization and a systematic overstatement of reported scores in many model cards, underscoring the need for transparent, domain-aware evaluation and reporting. The work offers practical guidelines for users and contributors to improve model selection and documentation practices, promoting reproducibility and reliable transfer to downstream tasks.

Abstract

With the massive surge in ML models on platforms like Hugging Face, users often lose track and struggle to choose the best model for their downstream tasks, frequently relying on model popularity indicated by download counts, likes, or recency. We investigate whether this popularity aligns with actual model performance and how the comprehensiveness of model documentation correlates with both popularity and performance. In our study, we evaluated a comprehensive set of 500 Sentiment Analysis models on Hugging Face. This evaluation involved massive annotation efforts, with human annotators completing nearly 80,000 annotations, alongside extensive model training and evaluation. Our findings reveal that model popularity does not necessarily correlate with performance. Additionally, we identify critical inconsistencies in model card reporting: approximately 80% of the models analyzed lack detailed information about the model, training, and evaluation processes. Furthermore, about 88% of model authors overstate their models' performance in the model cards. Based on our findings, we provide a checklist of guidelines for users to choose good models for downstream tasks.

Model Hubs and Beyond: Analyzing Model Popularity, Performance, and Documentation

TL;DR

The paper investigates whether popularity signals on Hugging Face align with real model performance and documentation quality by evaluating the top sentiment-analysis models across three benchmarks and a Reddit dataset. It combines large-scale manual inspection of model cards (approximately elements) with a dataset cartography framework to assess performance across Easy/Ambiguous/Hard instances, revealing weak correlations between popularity and performance and substantial documentation gaps. A case study on Reddit demonstrates poor generalization and a systematic overstatement of reported scores in many model cards, underscoring the need for transparent, domain-aware evaluation and reporting. The work offers practical guidelines for users and contributors to improve model selection and documentation practices, promoting reproducibility and reliable transfer to downstream tasks.

Abstract

With the massive surge in ML models on platforms like Hugging Face, users often lose track and struggle to choose the best model for their downstream tasks, frequently relying on model popularity indicated by download counts, likes, or recency. We investigate whether this popularity aligns with actual model performance and how the comprehensiveness of model documentation correlates with both popularity and performance. In our study, we evaluated a comprehensive set of 500 Sentiment Analysis models on Hugging Face. This evaluation involved massive annotation efforts, with human annotators completing nearly 80,000 annotations, alongside extensive model training and evaluation. Our findings reveal that model popularity does not necessarily correlate with performance. Additionally, we identify critical inconsistencies in model card reporting: approximately 80% of the models analyzed lack detailed information about the model, training, and evaluation processes. Furthermore, about 88% of model authors overstate their models' performance in the model cards. Based on our findings, we provide a checklist of guidelines for users to choose good models for downstream tasks.

Paper Structure

This paper contains 30 sections, 16 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Experiments
  4. Models Filtering Criteria
  5. Model Card Inspection
  6. Datasets
  7. Training Strategies
  8. Evaluate
  9. Finetune
  10. Training and Evaluation Methodology
  11. Evaluation Metrics
  12. Results and Discussions
  13. Does High Popularity Necessarily Correlate With Better Model Performance?
  14. Manual Inspection of $M_{500}$
  15. Does the Performance of a Model Correlate With How Well Documented Its Model Cards Are?
  16. ...and 15 more sections

Figures (16)

  • Figure 1: An illustration of a Model Card
  • Figure 2: Distribution of F1 scores for models in $M_{500}$ across different difficulty categories in dataset $\mathcal{D}$, under both the Evaluate and Finetune strategies.
  • Figure 3: Results of manual inspection of model cards as described in §\ref{['sec:resource']}. Each pie chart corresponds to a specific section within the model card.
  • Figure 4: Results of manual inspection of class labels for the $M_{500}$ models.
  • Figure 5: Results of manual inspection of model cards (granular analysis), as detailed in §\ref{['sec:resource']}. Each bar corresponds to a specific element within a section of the model card.
  • ...and 11 more figures