Big Reasoning with Small Models: Instruction Retrieval at Inference Time

TL;DR

Small language models struggle with multi-step reasoning and domain knowledge, but this work shows that inference-time instruction retrieval can bridge the gap to larger models. The authors build an Instruction Corpus by clustering training examples and generating modular, two-part instructions with GPT-5, then retrieve top-k instructions to guide decoding across MedQA, MMLU Law, and MathQA without fine-tuning. Across 3B–14B models, retrieved instructions yield consistent gains, especially on knowledge-intensive tasks, with concise prompts outperforming verbose ones and model-family effects often dominating over size. The approach preserves privacy and efficiency by externalizing reasoning in a non-parametric corpus, and results motivate competence-aware retrieval for adaptive, on-device reasoning.

Abstract

Can we bring large-scale reasoning to local-scale compute? Small language models (SLMs) are increasingly attractive because they run efficiently on local hardware, offering strong privacy, low cost, and reduced environmental impact. Yet they often struggle with tasks that require multi-step reasoning or domain-specific knowledge. We address this limitation through instruction intervention at inference time, where an SLM retrieves structured reasoning procedures rather than generating them from scratch. Our method builds an Instruction Corpus by grouping similar training questions and creating instructions via GPT-5. During inference, the SLM retrieves the most relevant instructions and follows their steps. Unlike retrieval-augmented generation, which retrieves text passages, instruction retrieval gives the model structured guidance for reasoning. We evaluate this framework on MedQA (medical board exams), MMLU Professional Law, and MathQA using models from 3B to 14B parameters without any additional fine-tuning. Instruction retrieval yields consistent gains: 9.4% on MedQA, 7.9% on MMLU Law, and 5.1% on MathQA. Concise instructions outperform longer ones, and the magnitude of improvement depends strongly on model family and intrinsic reasoning ability.

Figures (13)

• Figure 1: Comparison of reasoning steps across model configurations on a MedQA-style case. The small language model alone produces an incorrect causal chain, while a large model reaches the right diagnosis but at high computational and privacy cost. Adding retrieved instructions supplies the missing background and procedural steps, enabling the SLM to reproduce the correct reasoning efficiently and privately.
• Figure 2: Accuracy of SLMs with and without instruction retrieval across three reasoning benchmarks. Each pair shows zero-shot accuracy (black) and performance with High-School Concise instructions (green = improvement, red = decline). The dashed blue line marks GPT-4o’s zero-shot accuracy. Instruction retrieval yields consistent gains once models exceed 3B parameters, especially on MedQA and MMLU (Law), where knowledge and procedural reasoning are most demanding. Appendix Table \ref{['tab:aggregate-hs-concise']} reports the same results in tabular form for reference.
• Figure 3: Accuracy differences when enforcing specific instruction styles compared to baseline instructions with no style or length constraints. Concise instructions generally outperform the baseline, while verbose instructions tend to reduce accuracy. Audience level (High School vs. Graduate) has smaller and less consistent effects.
• Figure 4: Marginal effects on accuracy relative to zero-shot prompting. All instruction variants yield large gains (+28–29pp). Family effects dominate: Qwen3 (+56pp) and DeepSeek R1 (+38–43pp) show the strongest ability to follow instructions compared to the LLaMA-3 reference, while Gemma-2 (+16pp) and Mistral (+2pp) are smaller. Model scale adds a consistent +9pp per log-unit increase in parameters, while longer prompts reduce accuracy by –8pp per log-token.
• Figure 5: Performance impact of ablating instruction components on MedQA (High School Concise). Both knowledge-only and reasoning-only variants outperform zero-shot prompting, though the relative advantage differs by model family.