Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation

TL;DR

Small Language Models (SLMs) aim to deliver competitive NLP performance under tight resource constraints. The paper surveys architectures, training strategies, and compression techniques that enable compact models, and proposes a taxonomy for optimization and evaluation. It covers lightweight transformers, efficient self-attention, NAS, and multi-modal integration, as well as pruning, quantization, and knowledge transfer, with detailed discussion of datasets and metrics. The authors also address remaining challenges—hallucination, bias, energy-efficient inference, and data privacy—and outline directions for future research, highlighting practical implications for edge devices and privacy-preserving deployment.

Abstract

Small Language Models (SLMs) have gained substantial attention due to their ability to execute diverse language tasks successfully while using fewer computer resources. These models are particularly ideal for deployment in limited environments, such as mobile devices, on-device processing, and edge systems. In this study, we present a complete assessment of SLMs, focussing on their design frameworks, training approaches, and techniques for lowering model size and complexity. We offer a novel classification system to organize the optimization approaches applied for SLMs, encompassing strategies like pruning, quantization, and model compression. Furthermore, we assemble SLM's studies of evaluation suite with some existing datasets, establishing a rigorous platform for measuring SLM capabilities. Alongside this, we discuss the important difficulties that remain unresolved in this sector, including trade-offs between efficiency and performance, and we suggest directions for future study. We anticipate this study to serve as a beneficial guide for researchers and practitioners who aim to construct compact, efficient, and high-performing language models.