Privileged Information Distillation for Language Models

TL;DR

This paper introduces two training-time privileged information (PI) methods, Privileged Information Distillation (π-Distill) and On-Policy Self-Distillation (OPSD), to transfer PI-derived capabilities from frontier models to test-time policies that operate without PI in multi-turn tool-using environments. Both approaches share parameters and leverage a PI-conditioned teacher to guide learning, either through joint teacher-student optimization (π-Distill) or on-policy RL with a reverse KL penalty (OPSD). Across Travel Planner, τ-Bench, and GEM-based OOD benchmarks, π-Distill and OPSD consistently outperform standard SFT+RL baselines that assume access to full CoT traces, with π-Distill often delivering the strongest gains and OPSD offering a robust alternative when CoT is unavailable. The results illuminate key factors that enable effective PI transfer, notably the information content of PI and the balance between teacher and student learning, and reveal favorable scaling and generalization to unseen domains. Overall, the work demonstrates that training-time PI can be leveraged to produce high-performing policies without requiring CoT at test time, with broad implications for deploying sophisticated frontier-model reasoning in real-world, constrained settings.

Abstract

Training-time privileged information (PI) can enable language models to succeed on tasks they would otherwise fail, making it a powerful tool for reinforcement learning in hard, long-horizon settings. However, transferring capabilities learned with PI to policies that must act without it at inference time remains a fundamental challenge. We study this problem in the context of distilling frontier models for multi-turn agentic environments, where closed-source systems typically hide their internal reasoning and expose only action trajectories. This breaks standard distillation pipelines, since successful behavior is observable but the reasoning process is not. For this, we introduce π-Distill, a joint teacher-student objective that trains a PI-conditioned teacher and an unconditioned student simultaneously using the same model. Additionally, we also introduce On-Policy Self-Distillation (OPSD), an alternative approach that trains using Reinforcement Learning (RL) with a reverse KL-penalty between the student and the PI-conditioned teacher. We show that both of these algorithms effectively distill frontier agents using action-only PI. Specifically we find that π-Distill and in some cases OPSD, outperform industry standard practices (Supervised finetuning followed by RL) that assume access to full Chain-of-Thought supervision across multiple agentic benchmarks, models, and forms of PI. We complement our results with extensive analysis that characterizes the factors enabling effective learning with PI, focusing primarily on π-Distill and characterizing when OPSD is competitive.

Figures (17)

• Figure 1: Overview of the $\pi$-Distill framework. (1) Successful trajectories (not shown) are collected from a frontier agent that exposes only actions while hiding its Chain-of-Thought. (2) These trajectories are transformed into training-time privileged information (PI) and used to sample a PI-conditioned teacher policy $\textcolor{Teacher}{\pi^{\mathrm{T}}_\theta(\mathbf{o}\mid s, \mathbf{I})}$. (3) The PI-conditioned teacher and an unconditioned student $\textcolor{Student}{\pi^{S}_\theta(\mathbf{o}|s)}$ share parameters and are trained jointly, enabling transfer of privileged knowledge to a test-time policy that acts without PI.
• Figure 2: Results for Qwen3-8B on TravelPlanner and $\tau$-Bench retail. The dashed line separating SFT w/ CoT + RL denotes that this method is not a required baseline, as all other methods do not rely on frontier-model CoT traces. We find that both$\pi$-Distill and OPSD substantially outperform all baselines in this setting.
• Figure 3: Deriving PI from frontier model traces. The left panel illustrates sampling trajectories from a closed-source frontier model, where full CoT reasoning is typically occluded openai2024openaio1card. We transform these raw traces into three types of PI with varying information density: (1) Tool Calls & Arguments, retaining the exact actions invoked by the frontier model; (2) Tool Calls Only, where arguments are stripped; and (3) Self-Generated Hints, where the student model summarizes the frontier trajectory into a concise hint.
• Figure 4: Evaluation on Out-of-Domain Environments. We report Pass@1 and Pass@10 on the GEM search-tool benchmark suite (7 datasets) for Qwen 3 models and R1-Distill-Llama-8B, using the best checkpoint selected on $\tau$-Bench Retail. Bars show mean $\pm$ standard errors over three seeds per dataset, comparing $\pi$-Distill variants ($\textcolor{Teacher}{\pi^{\mathrm{T}}}$ ($\alpha=1$), $\textcolor{Student}{\pi^{S}}$ ($\alpha=0$), $\textcolor{Teacher}{\pi^{\mathrm{T}}} + \textcolor{Student}{\pi^{S}}$ ($\alpha=0.5$)) and OPSD against SFT w/ CoT + RL, $\pi_{\text{Base}}$, and standard RL. The dashed line separating SFT w/ CoT + RL denotes that this method is not considered a required baseline, as all PI methods avoid relying on frontier-model CoT traces. We consistently find that both algorithms exhibit substantially less forgetting than standard RL. Moreover, we find $\pi$-Distill and OPSD generalize significantly better than SFT w/ CoT + RL when using Qwen 3-8B.
• Figure 5: Impact of PI Types and Algorithms on Performance. We compare held-out performance on $\tau$-Bench (top row) and Travel Planner (bottom row) across three base models and three PI types (colors). The scatter plots map final scores against the initial teacher-student divergence ($D_{\mathrm{KL}}(\pi^T_{\text{base}} \,\|\, \pi^S_{\text{base}})$), while the bar-charts display the PI utility ($\Delta$) on training tasks. Key observations: (1) Higher initial KL divergence generally correlates with decreased final performance. (2) Joint training ($\alpha=0.5$, $\triangle$) is the most stable configuration, performing best in 6/16 scenarios and worst in a single one. (3) Student-only training ($\alpha=0$, $\square$) requires low KL and positive utility (note the failure in Planner QWEN3 8B where $\Delta < 0$). Conversely, Teacher-only training ($\alpha=1$, $\bigcirc$) degrades as KL increases or fails due to policy collapse when KL is negligible.