Targeted Error Correction in Knowledge Distillation: Small Language Models Surpass GPT

TL;DR

The paper tackles the high cost and privacy risks of proprietary LLMs in production by introducing the Analyze-Revise-Finetune (ARF) pipeline. ARF first analyzes common errors in teacher-generated summaries, uses a smaller editor LLM to perform targeted corrections, and then fine-tunes a compact student LLM on the revised data. The approach, demonstrated on eBay's customer-service summarization task, shows that open-source Llama 3.1 8B and 70B variants can surpass GPT-3.5 with r1 revisions, while reducing cost and preserving data privacy. This supports a generalizable, privacy-conscious pathway to elevate small LLMs across diverse applications.

Abstract

We introduce an Analyze-Revise-Finetune (ARF) pipeline that enables smaller open-source language models (LLMs) to surpass substantially larger proprietary models in customer service summarization tasks. The pipeline first analyzes and categorizes common errors in summaries produced by a teacher model (GPT-3.5), then performs a targeted revision using a compact editor model (Llama 3.1 70B) to generate high-quality, refined training data. Fine-tuning a smaller student model (Llama 3.1 8B) on this refined data resulted in superior summarization performance compared to GPT-3.5. The ARF pipeline improves cost efficiency and data privacy while maintaining competitive accuracy, illustrating a generalizable framework for enhancing open-source LLMs across diverse downstream applications.

Figures (2)

• Figure 1: "Analyze-Revise-Finetune" pipeline
• Figure 2: Prompts for summary error correction