$\textit{SKIntern}$: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models

TL;DR

SKIntern tackles the challenge of imparting rich reasoning ability to Small Language Models without the heavy cost of external retrieval. It uses a curriculum-inspired progressive fine-tuning regime that gradually internalizes symbolic knowledge and few-shot examples into model parameters, controlled by a linear decay schedule. Empirical results on open-source SLMs across factual, mathematical, and general reasoning benchmarks show that SKIntern achieves over $5\%$ gains and up to $4\times$ FLOP reductions on ID and OOD tasks, surpassing Std-CoT, MT-CoT, KARD, and CasCoD baselines. The approach promises practical efficiency gains for cost-sensitive deployment and suggests avenues for scaling with larger LLM teachers and broader task sets.

Abstract

Small Language Models (SLMs) are attracting attention due to the high computational demands and privacy concerns of Large Language Models (LLMs). Some studies fine-tune SLMs using Chains of Thought (CoT) data distilled from LLMs, aiming to enhance their reasoning ability. Furthermore, Some CoT distillation methods introduce external symbolic knowledge into the generation process to improve the limited knowledge memory, reasoning ability and out-of-domain (OOD) generalization of SLMs. However, the introduction of symbolic knowledge increases computational overhead and introduces potential noise. In this paper, we introduce $\textit{SKIntern}$, an innovative approach that empowers SLMs to internalize symbolic knowledge and few-shot examples gradually through a progressive fine-tuning process, guided by a predefined linear decay schedule under curriculum learning. By efficiently internalizing knowledge, $\textit{SKIntern}$ reduces computational overhead and speeds up the reasoning process by focusing solely on the question during inference. It outperforms state-of-the-art baselines by over 5\%, while reducing inference costs (measured in FLOPs) by up to $4\times$ across a wide range of SLMs in both in-domain (ID) and out-of-domain (OOD) tasks. Our code will be available at \url{https://github.com/Xnhyacinth/SKIntern}.

Figures (5)

• Figure 1: Knowledge utilization comparisons of SKIntern and other typical CoT distillation methods. (i) Std-CoT: SLM is fine-tuned to generate the rationale and answer for the question (Q -> R + A). (ii) KARD: Fine-tune the SLM to generate the rationale and answer based on the question and the retrieved symbolic knowledge (Q + K -> R + A). (iii): CasCoD: Decompose the single CoT learning step into two comprehensive learning steps of rationale generation (Q -> R) and rationale utilization (Q + R -> A). (iv): SKIntern: Like human interns, SLMs gradually absorb and internalize symbolic knowledge provided by LLMs during the progressive fine-tuning, thereby achieving efficient (Q -> R + A) and effective reasoning (modeling K in parameters).
• Figure 2: Overview of the SKIntern framework. SKIntern starts with full symbolic knowledge and examples, and progressively prunes them to gradually internalize knowledge, reducing the prompt length and the number of computations towards the SLM. Based on schedule $\mathcal{S}$, we perform effective knowledge compression and example pruning before fine-tuning the SLM to generate rationales and answers. Gradual fine-tuning makes SLMs internalize knowledge and examples into parameters, thereby enhancing performance without increasing computational cost.
• Figure 3: Accuracy (%) against FLOPs for varying model sizes. FLOPs calculations are based on processing all examples from the same task during inference.
• Figure 4: Efficiency on training data and model size. The backbone model for the data size variation is Qwen2-7B.
• Figure 5: Ablation studies of $\boldsymbol{k}$ on vanilla methods.