Didactic to Constructive: Turning Expert Solutions into Learnable Reasoning

TL;DR

This work addresses the difficulty of teaching frontier language models to reason on highly challenging problems when reward signals are scarce. It introduces Distribution Aligned Imitation Learning (DAIL), a two-stage method that first expands expert, didactic solutions into in-distribution reasoning traces via a privileged student and mixed policy rollouts, then trains the student with a contrastive objective against a negative reference to discourage rationalization shortcuts. Using fewer than 1000 high-quality expert solutions, DAIL achieves 10-25% gains in pass@k on math benchmarks, doubles to quadruples reasoning efficiency, and generalizes to out-of-domain tasks, including GPQA. The approach demonstrates that learning from expert reasoning can be made viable for difficult, non-verifiable problems, potentially widening the practical impact of large reasoning models in domains with sparse reward signals.

Abstract

Improving the reasoning capabilities of large language models (LLMs) typically relies either on the model's ability to sample a correct solution to be reinforced or on the existence of a stronger model able to solve the problem. However, many difficult problems remain intractable for even current frontier models, preventing the extraction of valid training signals. A promising alternative is to leverage high-quality expert human solutions, yet naive imitation of this data fails because it is fundamentally out of distribution: expert solutions are typically didactic, containing implicit reasoning gaps intended for human readers rather than computational models. Furthermore, high-quality expert solutions are expensive, necessitating generalizable sample-efficient training methods. We propose Distribution Aligned Imitation Learning (DAIL), a two-step method that bridges the distributional gap by first transforming expert solutions into detailed, in-distribution reasoning traces and then applying a contrastive objective to focus learning on expert insights and methodologies. We find that DAIL can leverage fewer than 1000 high-quality expert solutions to achieve 10-25% pass@k gains on Qwen2.5-Instruct and Qwen3 models, improve reasoning efficiency by 2x to 4x, and enable out-of-domain generalization.

Figures (16)

• Figure 1: Overview of DAIL. Starting from a small set of expert solutions, we generate in-distribution transformed solutions for training via mixed policy decoding: the student model uses the expert solution as a reference to produce a detailed reasoning trace. This process mitigates didactic shortcuts, e.g., the expert solution skips the proof of why $\angle AOM$ is acute (in red), while the transformed solution explicitly details this reasoning (in green). Using this dataset of transformed solutions, DAIL applies contrastive learning on paired full vs. partial solutions to discourage imitating rationalization shortcuts in transformed solutions, e.g., the model ignoring the irrational part of a derived result to force the generation of the correct answer (in orange).
• Figure 2: Pass@$k$ performance comparison of DAIL on Qwen2.5-7B-Instruct with e1-verifiable compared to RLVR methods on IMO-Answer, Beyond AIME, and AIME 2024 / 2025 benchmarks. DAIL exhibits consistent performance improvements over the base instruction model, while applying RLVR methods results in pass@k reductions due to the difficulty of training dataset problems.
• Figure 3: Baselines. Comparing pass@k performance of DAIL to temperature, STaR rationalization, and direct SFT on expert solutions baselines. To better visualize the relative performance between baselines, the results are plotted for $k \ge 8$. See Figure \ref{['fig:all_naive_baselines']} for results at lower $k$ values.
• Figure 4: Test-Time Efficiency. Performance on mathematics benchmarks under various token reasoning limits. To ensure gains are not simply due to non-response, models are given 2048 tokens beyond these limits to produce their final answer. Compared to Qwen3-8B (think), the model trained with DAIL on e1-proof yields improved performance across token budgets.
• Figure 5: Contrastive loss consistently outperforms NLL. Comparison of the performance of Qwen2.5-7B-Instruct trained with DAIL's contrastive objective and standard NLL. Contrastive loss outperforms NLL across generation settings and metrics.