Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

Sanket Badhe, Deep Shah

Abstract

Advanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative, fine-tuning smaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introduce Prompt-Level Distillation (PLD). We extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model's System Prompt. Evaluated on the StereoSet and Contract-NLI datasets using Gemma-3 4B, PLD improved Macro F1 scores from 57\% to 90.0\% and 67\% to 83\% respectively, enabling this compact model to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.

Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning

Abstract

Advanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative, fine-tuning smaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introduce Prompt-Level Distillation (PLD). We extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model's System Prompt. Evaluated on the StereoSet and Contract-NLI datasets using Gemma-3 4B, PLD improved Macro F1 scores from 57\% to 90.0\% and 67\% to 83\% respectively, enabling this compact model to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.
Paper Structure (31 sections, 3 figures, 2 tables)

This paper contains 31 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain-of-Thought and Inference Efficiency
  4. Knowledge Distillation and Reasoning Transfer
  5. Automatic Prompt Optimization (APO)
  6. Methodology
  7. Phase 1: Supervised Instruction Extraction
  8. Phase 2: Clustering Logic Synthesis
  9. Phase 3: Conflict Resolution
  10. Phase 4: Inference
  11. Experimental Setup
  12. Setup
  13. Baseline
  14. Datasets
  15. Results and Analysis
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Overview of Prompt-Level Distillation (PLD). The framework operates in four phases: (1) Supervised Instruction Extraction from training data, (2) Semantic Synthesis using clustering, (3) A Closed-Loop Conflict Resolution phase to refine logic, and (4) Zero-shot Inference using the consolidated system prompt.
  • Figure 2: E2E Model Latency between Gemma-3 4B, Gemini 2 Flash, and Gemini 3 Flash
  • Figure 3: Comparision of Model pricing