MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimization

TL;DR

MetaGDPO tackles catastrophic forgetting when distilling reasoning from large LLMs to smaller models by coupling metacognitive knowledge-driven data construction with a resource-efficient GDPO training objective. The framework introduces MetaKL, a metacognitively annotated dataset of ~5K prompts that aligns training data with base-model abilities, and a Group Direct Preference Optimization method that leverages offline high-quality responses and reduces gradient-computation from $O(G^2)$ to $O(G)$. Across 12 benchmarks and multiple model families, MetaGDPO improves reasoning performance on smaller models while preserving prior capabilities, achieving relative gains of roughly 5–10% on average over baselines. The approach demonstrates a practical, data-informed, and scalable path to deploy compact models with strong reasoning across math, general reasoning, and safety domains.

Abstract

Large Language Models demonstrate strong reasoning capabilities, which can be effectively compressed into smaller models. However, existing datasets and fine-tuning approaches still face challenges that lead to catastrophic forgetting, particularly for models smaller than 8B. First, most datasets typically ignore the relationship between training data knowledge and the model's inherent abilities, making it difficult to preserve prior knowledge. Second, conventional training objectives often fail to constrain inherent knowledge preservation, which can result in forgetting of previously learned skills. To address these issues, we propose a comprehensive solution that alleviates catastrophic forgetting from both the data and fine-tuning approach perspectives. On the data side, we construct a dataset of 5K instances that covers multiple reasoning tasks and incorporates metacognitive knowledge, making it more tolerant and effective for distillation into smaller models. We annotate the metacognitive knowledge required to solve each question and filter the data based on task knowledge and the model's inherent skills. On the training side, we introduce GDPO (Group Direction Preference Optimization), which is better suited for resource-limited scenarios and can efficiently approximate the performance of GRPO. Guided by the large model and by implicitly constraining the optimization path through a reference model, GDPO enables more effective knowledge transfer from the large model and constrains excessive parameter drift. Extensive experiments demonstrate that our approach significantly alleviates catastrophic forgetting and improves reasoning performance on smaller models.

Figures (10)

• Figure 1: Small-scale fine-tuning on challenging mathematical data such as LIMO can enhance the performance of 32B models, whereas 8B models tend to suffer from severe performance degradation.
• Figure 2: The framework of our paper. We first construct the data based on metacognitive knowledge with analyzing base models' inherent ability. Then, we utilize the Group Direct Preference Optimization to fine-tune the base model.
• Figure 3: The performance on metacognitive knowledge of base models.
• Figure 4: The performance varies with the number of groups and training data size.
• Figure 5: Comparison of model performance across different training data compositions on Qwen3-8B.

Theorems & Definitions (1)

• proof