Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What is the Role of Small Models in the LLM Era: A Survey

Lihu Chen, Gaël Varoquaux

TL;DR

<3-5 sentence high-level summary>This survey investigates the role of Small Models (SMs) in the era of Large Language Models (LLMs) by analyzing how SMs can collaborate with LLMs to improve efficiency, robustness, and interpretability, and how LLMs can, in turn, enhance SMs via distillation and data synthesis. It presents a dual framework of Collaboration and Competition, detailing concrete methods in data curation, efficient inference, augmented reasoning, distillation, and synthetic data generation, as well as deployment considerations across computation-, task-, and interpretability-constrained environments. The work emphasizes practical strategies—such as data-quality focused curation, model cascading/routing, Retrieval Augmented Generation, and weak-to-strong supervision—that enable resource-constrained practitioners to deploy effective AI systems. By outlining actionable directions and pitfalls, it highlights the ecological niche of SMs as cost-effective, interpretable allies that synergize with LLMs to enable scalable, responsible AI deployments.

Abstract

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models

What is the Role of Small Models in the LLM Era: A Survey

TL;DR

<3-5 sentence high-level summary>This survey investigates the role of Small Models (SMs) in the era of Large Language Models (LLMs) by analyzing how SMs can collaborate with LLMs to improve efficiency, robustness, and interpretability, and how LLMs can, in turn, enhance SMs via distillation and data synthesis. It presents a dual framework of Collaboration and Competition, detailing concrete methods in data curation, efficient inference, augmented reasoning, distillation, and synthetic data generation, as well as deployment considerations across computation-, task-, and interpretability-constrained environments. The work emphasizes practical strategies—such as data-quality focused curation, model cascading/routing, Retrieval Augmented Generation, and weak-to-strong supervision—that enable resource-constrained practitioners to deploy effective AI systems. By outlining actionable directions and pitfalls, it highlights the ecological niche of SMs as cost-effective, interpretable allies that synergize with LLMs to enable scalable, responsible AI deployments.

Abstract

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models
Paper Structure (47 sections, 11 figures, 1 table)

This paper contains 47 sections, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Accuracy
  3. Generality
  4. Efficiency
  5. Interpretability
  6. Related Work
  7. Collaboration
  8. Small Models Enhance LLMs
  9. Data Curation
  10. Curating Pre-training Data
  11. Curating Fine-Tuning Data
  12. Summary and Future Directions
  13. Efficient Inference
  14. Model Cascading
  15. Model Routing
  16. ...and 32 more sections

Figures (11)

  • Figure 1: The relationship between model size and monthly downloads. This analysis considers open-source NLP models hosted on HuggingFace and categorizes them into five size groups based on the number of parameters: [200M, 500M, 1B, 3B]. The data was collected on October 13, 2025.
  • Figure 2: Collaborations between LLMs and SMs
  • Figure 3: Taxonomy of data curation
  • Figure 4: Taxonomy of efficient inference
  • Figure 5: Taxonomy of augmented reasoning
  • ...and 6 more figures