Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification

TL;DR

The paper tackles whether small language models can rival large LLMs in zero-shot text classification. It conducts a large-scale empirical study covering 15 datasets and 72 models spanning 77M to 70B parameters, across encoder–decoder and decoder–only architectures, with varied prompting and scoring strategies. The results show that small models can match or exceed large-model performance on many tasks, and that architecture and instruction tuning often drive results more than sheer size. These findings suggest resource-efficient, small models can be viable for zero-shot labeling in certain domains, with broad implications for accessibility and cost. The work further provides an open-source repository detailing the methodology to enable reproducibility and further exploration in this space.

Abstract

This study is part of the debate on the efficiency of large versus small language models for text classification by prompting.We assess the performance of small language models in zero-shot text classification, challenging the prevailing dominance of large models.Across 15 datasets, our investigation benchmarks language models from 77M to 40B parameters using different architectures and scoring functions. Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts.We developed and shared a comprehensive open-source repository that encapsulates our methodologies. This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.

Figures (4)

• Figure 1: Performance Comparison of Different Model Sizes Across Datasets.
• Figure 2: Performance Variation Across Different Architectures.
• Figure 3: Performance Comparison between Instruction-Tuned models or not Across Datasets.
• Figure 4: Performance Comparison between Instruction-Tuned models or not, Across Model Architecture