Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position: Enough of Scaling LLMs! Lets Focus on Downscaling

Yash Goel, Ayan Sengupta, Tanmoy Chakraborty

TL;DR

The paper challenges the primacy of neural scaling laws for developing large language models by highlighting diminishing returns and substantial environmental costs. It argues for a shift toward downscaling, supported by evidence from small, efficient models and data-quality innovations, and presents a framework combining data pruning, domain-adaptive training, model compression, and ensemble methods. Central contributions include a Domain-Continual Pre-Training (D-CPT) law, a post-pruning loss model (P2 law), and a theoretical downscaling condition for ensemble benefits, all integrated into a practical downscaling pipeline. The work emphasizes sustainability and accessibility, proposing concrete strategies to achieve robust performance with substantially lower compute and carbon emissions while preserving domain-specific capabilities.

Abstract

We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.

Position: Enough of Scaling LLMs! Lets Focus on Downscaling

TL;DR

The paper challenges the primacy of neural scaling laws for developing large language models by highlighting diminishing returns and substantial environmental costs. It argues for a shift toward downscaling, supported by evidence from small, efficient models and data-quality innovations, and presents a framework combining data pruning, domain-adaptive training, model compression, and ensemble methods. Central contributions include a Domain-Continual Pre-Training (D-CPT) law, a post-pruning loss model (P2 law), and a theoretical downscaling condition for ensemble benefits, all integrated into a practical downscaling pipeline. The work emphasizes sustainability and accessibility, proposing concrete strategies to achieve robust performance with substantially lower compute and carbon emissions while preserving domain-specific capabilities.

Abstract

We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws have provided critical insights into performance improvements through increasing model and dataset size, we emphasize the significant limitations of this approach, particularly in terms of computational inefficiency, environmental impact, and deployment constraints. To address these challenges, we propose a holistic framework for downscaling LLMs that seeks to maintain performance while drastically reducing resource demands. This paper outlines practical strategies for transitioning away from traditional scaling paradigms, advocating for a more sustainable, efficient, and accessible approach to LLM development.
Paper Structure (25 sections, 2 theorems, 34 equations, 5 figures)

This paper contains 25 sections, 2 theorems, 34 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Neural Scaling Laws
  3. Scaling Laws of LLMs
  4. Scaling Laws Beyond LLMs
  5. Criticisms of Neural Scaling Laws
  6. Rise of Small and Efficient Language Models
  7. Data Quality Evolution
  8. Architectural Innovations
  9. Advancements in Task Specialization and Training Methodology
  10. Efficiency Optimizations
  11. Our Recommendations
  12. Downscaling Datasets for Better Knowledge Alignment
  13. Model Downscaling
  14. Ensemble of LLMs
  15. Why Downscaling Law Might Work?
  16. ...and 10 more sections

Key Result

Proposition 2.1

where $(K_1 + K_2)$ represents a compound constant that encapsulates all hardware, data center, and efficiency parameters. This shows that CO2 emissions scale linearly with both (i) the number of model parameters $(N)$, and (ii) the amount of training data $(D)$.

Figures (5)

  • Figure 1: The growth of the number of papers on scaling laws for neural models over the past eight years.
  • Figure 2: Test loss decreases logarithmically with model size, whereas the estimated training-time carbon emission increases linearly.
  • Figure 3: Test loss decreases logarithmically with training token size, whereas the estimated training-time carbon emission increases linearly.
  • Figure 4: Number of open-sourced SLMs (size $100M$-$5B$) developed over the years (statistics taken from lu_small_2024).
  • Figure 5: Our proposed framework for model downscaling with adaptive domain adaptation. We decompose an LLM pre-training into multiple stages -- (1) calibration for compressing LLM into multiple SLM chunks, along with aligned pre-training corpus, (2) active learning-based dataset pruning, for optimal data downscaling, (3) Continual training of SLMs on aligned corpus and (4) model ensemble for merging SLM chunks to recover the larger model with original parameter size.

Theorems & Definitions (2)

  • Proposition 2.1
  • Proposition 4.1