Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration

Yong Wu, Weihang Pan, Ke Li, Chen Binhui, Ping Li, Binbin Lin

TL;DR

The paper tackles the challenge of eliciting reliable reasoning in small language models when trained on expert reasoning templates designed for larger models. It introduces Dynamic Adaptation of Reasoning Trajectories (DART), a framework that estimates step-wise adaptability via solution simulation, employs imitation feasibility to selectively imitate expert steps, and enables outcome-aligned autonomous exploration when imitation is infeasible. The method is formalized with a step-wise adaptability score $f_t=Q(s_{<t},s_t)$ and an outcome-consistency constraint to anchor autonomous continuations, optimizing a distillation objective $L_{\text{DART}}$. Empirical results across diverse benchmarks and model scales show that DART improves reasoning generalization and data efficiency compared to static fine-tuning, demonstrating scalable reasoning alignment for resource-constrained models. Overall, DART provides a principled, capacity-aware approach to data adaptation for reasoning that reduces brittleness and enhances robustness under distribution shifts.

Abstract

Large language models (LLMs) have shown remarkable reasoning capabilities, yet aligning such abilities to small language models (SLMs) remains a challenge due to distributional mismatches and limited model capacity. Existing reasoning datasets, typically designed for powerful LLMs, often lead to degraded performance when directly applied to weaker models. In this work, we introduce Dynamic Adaptation of Reasoning Trajectories (DART), a novel data adaptation framework that bridges the capability gap between expert reasoning trajectories and diverse SLMs. Instead of uniformly imitating expert steps, DART employs a selective imitation strategy guided by step-wise adaptability estimation via solution simulation. When expert steps surpass the student's capacity -- signaled by an Imitation Gap -- the student autonomously explores alternative reasoning paths, constrained by outcome consistency. We validate DART across multiple reasoning benchmarks and model scales, demonstrating that it significantly improves generalization and data efficiency over static fine-tuning. Our method enhances supervision quality by aligning training signals with the student's reasoning capabilities, offering a scalable solution for reasoning alignment in resource-constrained models.

Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration

TL;DR

The paper tackles the challenge of eliciting reliable reasoning in small language models when trained on expert reasoning templates designed for larger models. It introduces Dynamic Adaptation of Reasoning Trajectories (DART), a framework that estimates step-wise adaptability via solution simulation, employs imitation feasibility to selectively imitate expert steps, and enables outcome-aligned autonomous exploration when imitation is infeasible. The method is formalized with a step-wise adaptability score and an outcome-consistency constraint to anchor autonomous continuations, optimizing a distillation objective . Empirical results across diverse benchmarks and model scales show that DART improves reasoning generalization and data efficiency compared to static fine-tuning, demonstrating scalable reasoning alignment for resource-constrained models. Overall, DART provides a principled, capacity-aware approach to data adaptation for reasoning that reduces brittleness and enhances robustness under distribution shifts.

Abstract

Large language models (LLMs) have shown remarkable reasoning capabilities, yet aligning such abilities to small language models (SLMs) remains a challenge due to distributional mismatches and limited model capacity. Existing reasoning datasets, typically designed for powerful LLMs, often lead to degraded performance when directly applied to weaker models. In this work, we introduce Dynamic Adaptation of Reasoning Trajectories (DART), a novel data adaptation framework that bridges the capability gap between expert reasoning trajectories and diverse SLMs. Instead of uniformly imitating expert steps, DART employs a selective imitation strategy guided by step-wise adaptability estimation via solution simulation. When expert steps surpass the student's capacity -- signaled by an Imitation Gap -- the student autonomously explores alternative reasoning paths, constrained by outcome consistency. We validate DART across multiple reasoning benchmarks and model scales, demonstrating that it significantly improves generalization and data efficiency over static fine-tuning. Our method enhances supervision quality by aligning training signals with the student's reasoning capabilities, offering a scalable solution for reasoning alignment in resource-constrained models.

Paper Structure

This paper contains 32 sections, 6 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Limitations of Supervised Imitation on Expert Trajectories
  3. Problem Definition: Reasoning Capability Elicitation via Minimal Demonstrations
  4. Limitations of Supervised Imitation on Expert Demonstrations
  5. The Need for Imitation Feasibility-Aware Adaptation.
  6. Methodology
  7. Step-wise Adaptability Estimation via Solution Simulation
  8. Adaptive Path Exploration
  9. Learning from Outcome-Aligned Adapted Trajectories
  10. Experiments
  11. Experimental Setup
  12. Adaptation Datasets.
  13. Adaptation Strategies.
  14. Benchmark Tasks.
  15. Main Results
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Overview of the DART framework.
  • Figure 2: Step-wise adaptability scores across expert trajectories for student models of varying sizes (0.5B, 1.5B, 3B parameters) under LIMO (top row) and Math-QwQ-32B dataset (bottom row) supervision. The emergence of the Imitation Gap is evident: initial steps yield positive adaptation, but continued step-by-step imitation can become harmful.
  • Figure 3: Repetition ratios(%) in search paths across different model sizes and datasets. Smaller models tend to have higher repetition ratios, particularly on the Limo dataset.