Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discovering Hidden Gems in Model Repositories

Jonathan Kahana, Eliahu Horwitz, Yedid Hoshen

TL;DR

The paper addresses the problem that model usage in public repositories is highly centralized despite a large long tail of potential contenders. It reframes model discovery as a budgeted Best Arm Identification problem using Multi-Armed Bandits and introduces two domain-specific enhancements—Correlated Sampling and an Aggressive Elimination schedule—within Sequential Halving. The proposed method reliably uncovers Hidden Gems across multiple model trees, achieving top-3 results with only about 50 queries per candidate and over 50x speedups compared to exhaustive search, with notable gains on tasks like GSM8K and RouterBench. This work demonstrates that better-performing models exist outside the popular set, offering a practical, scalable approach to more effective model selection in large repositories.

Abstract

Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.

Discovering Hidden Gems in Model Repositories

TL;DR

The paper addresses the problem that model usage in public repositories is highly centralized despite a large long tail of potential contenders. It reframes model discovery as a budgeted Best Arm Identification problem using Multi-Armed Bandits and introduces two domain-specific enhancements—Correlated Sampling and an Aggressive Elimination schedule—within Sequential Halving. The proposed method reliably uncovers Hidden Gems across multiple model trees, achieving top-3 results with only about 50 queries per candidate and over 50x speedups compared to exhaustive search, with notable gains on tasks like GSM8K and RouterBench. This work demonstrates that better-performing models exist outside the popular set, offering a practical, scalable approach to more effective model selection in large repositories.

Abstract

Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.
Paper Structure (19 sections, 8 figures, 11 tables)

This paper contains 19 sections, 8 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Do Hidden Gems Exist?
  3. Efficient Model Discovery
  4. Experiments
  5. Conclusion
  6. Limitations
  7. Evaluation Resources.
  8. Other Tasks.
  9. Related Works and Model Populations
  10. Implementation Details
  11. Model Selection.
  12. Query Selection.
  13. Model Evaluation.
  14. Scheduler.
  15. Ablation Studies
  16. ...and 4 more sections

Figures (8)

  • Figure 1: Hidden Gems & Repository Inefficiency.Center: The Llama3.1-8B Model Tree, where node size reflects downloads (log scale). Our evaluation reveals "hidden gems" (circled): unpopular models that significantly outperform widely used baselines. Top Left: Cumulative download rates showing usage is extremely concentrated in a tiny fraction of top models. Bottom Right: The vast majority of models (over $90\%$) are rarely explored, receiving $\leq15$ monthly downloads.
  • Figure 2: Visualizing Hidden Gems in Model Trees. Nodes represent models, with size corresponding to monthly downloads in log-scale. (a) Qwen-3B tree colored by MBPP$_s$ coding performance; the best coder (circled) remains unpopular despite significantly outperforming the base instruct model. (b) Qwen-7B tree highlighting gems in math, coding and overall (RouterBench$_s$) performance. (c) Llama-3.1-8B tree colored by overall performance; the best-performing model has orders of magnitude fewer downloads than the official Llama3.1 8B Instruct version.
  • Figure 3: Our proposed Model Search Algorithm
  • Figure 4: Cumulative Accuracy Distributions across different model architectures and specialized tasks.
  • Figure 5: Different performance views for the Qwen-3B model tree.
  • ...and 3 more figures