Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?

TL;DR

The paper investigates how small language models (SLMs) learn, unlearn, and retain noise patterns embedded in prompts, focusing on five noise types (word flip, character flip, transliteration, irrelevant content, and counterfactual information). Using four SLMs under 3B parameters and larger LLM baselines, the study builds a clean instruction-tuning corpus and multiple noise-augmented datasets, then evaluates behavior across learning, unlearning, and retention phases with semantic similarity metrics and LLM-driven judgments. Key findings show substantial variability driven by model pretraining quality and noise type: Phi2 exhibits robustness to several noise categories, Gemma handles transliteration well, while Olmo is more susceptible to learning noise; importantly, all models can effectively unlearn noise patterns after subsequent clean-data training, with negligible retention. The results imply practical noise-mitigation strategies for real-world deployment, emphasizing data quality and targeted training sequences over mere model scale, and provide a blueprint for robustness in edge-friendly SLMs.

Abstract

With the growing need for efficient language models in resource-constrained environments, Small Language Models (SLMs) have emerged as compact and practical alternatives to Large Language Models (LLMs). While studies have explored noise handling in LLMs, little is known about how SLMs handle noise, a critical factor for their reliable real-world deployment. This study investigates the ability of SLMs with parameters between 1 and 3 billion to learn, retain, and subsequently eliminate different types of noise (word flip, character flip, transliteration, irrelevant content, and contradictory information). Four pretrained SLMs (Olmo 1B, Qwen1.5 1.8B, Gemma1.1 2B, and Phi2 2.7B) were instruction-tuned on noise-free data and tested with in-context examples to assess noise learning. Subsequently, noise patterns were introduced in instruction tuning to assess their adaptability. The results revealed differences in how models handle noise, with smaller models like Olmo quickly adapting to noise patterns. Phi2's carefully curated, structured, and high-quality pretraining data enabled resistance to character level, transliteration, and counterfactual noise, while Gemma adapted successfully to transliteration noise through its multilingual pretraining. Subsequent clean data training effectively mitigated noise effects. These findings provide practical strategies for developing robust SLMs for real-world applications.

Figures (3)

• Figure 1: Tokenization of “Science fiction".
• Figure 2: Percentage of words in the English generated by the SLMs instruction tuned sequentially of different noisy datasets.
• Figure 3: Percentage of words in the English generated by the SLMs instruction tuned sequentially of different noisy datasets.