Privileged Information Distillation for Language Models
Training-time privileged information (PI) can enable language models to succeed on tasks they would otherwise fail, making it a powerful tool for reinforcement learning in hard, long-horizon settings. However, transferring capabilities learned with PI to policies that must act without it at inference time remains a fundamental challenge. We study this problem in the context of distilling frontier models for multi-turn agentic environments, where closed-source systems typically hide their internal reasoning and expose only action trajectories. This breaks standard distillation pipelines, since successful behavior is observable but the reasoning process is not. For this, we introduce π-Distill, a joint teacher-student objective that trains a PI-conditioned teacher and an unconditioned student simultaneously using the same model. Additionally, we also introduce On-Policy Self-Distillation (OPSD), an alternative approach that trains using Reinforcement Learning (RL) with a reverse KL-penalty between the student and the PI-conditioned teacher. We show that both of these algorithms effectively distill frontier agents using action-only PI. Specifically we find that π-Distill and in some cases OPSD, outperform industry standard practices (Supervised finetuning followed by RL) that assume access to full Chain-of-Thought supervision across multiple agentic benchmarks, models, and forms of PI. We complement our results with extensive analysis that characterizes the factors enabling effective learning with PI, focusing primarily on π-Distill and characterizing when OPSD is competitive.
6985f5fc30e925bc620acf9f
Emiliano Penaloza
6985f5fc30e925bc620acfa0
Dheeraj Vattikonda
claimed_verified
2026-02-06T18:49:44.400Z
65fb7b1c876d4642e8340e95
/avatars/c258c4528315243b98c1e60ba398e98b.svg
Dheeraj Vattikonda
Dheeraj46329
user
Dheeraj46329
6985f5fc30e925bc620acfa1
Nicolas Gontier
6985f5fc30e925bc620acfa2
Alexandre Lacoste
6985f5fc30e925bc620acfa3
Laurent Charlin
6985f5fc30e925bc620acfa4
Massimo Caccia
claimed_verified
2026-02-06T18:49:46.321Z
64b04320e5000ae8a57570a7
/avatars/226f57ea23b9dd861e71b7ba7821adce.svg
Massimo Caccia
optimass
user
optimass
Privileged Information Distillation for Language Models
Training-time privileged information (PI) can enable language models to succeed on tasks they would otherwise fail, making it a powerful tool for reinforcement learning in hard, long-horizon settings. However, transferring capabilities learned with PI to policies that must act without it at inference time remains a fundamental challenge. We study this problem in the context of distilling frontier models for multi-turn agentic environments, where closed-source systems typically hide their internal reasoning and expose only action trajectories. This breaks standard distillation pipelines, since successful behavior is observable but the reasoning process is not. For this, we introduce π-Distill, a joint teacher-student objective that trains a PI-conditioned teacher and an unconditioned student simultaneously using the same model. Additionally, we also introduce On-Policy Self-Distillation (OPSD), an alternative approach that trains using Reinforcement Learning (RL) with a reverse KL-penalty between the student and the PI-conditioned teacher. We show that both of these algorithms effectively distill frontier agents using action-only PI. Specifically we find that π-Distill and in some cases OPSD, outperform industry standard practices (Supervised finetuning followed by RL) that assume access to full Chain-of-Thought supervision across multiple agentic benchmarks, models, and forms of PI. We complement our results with extensive analysis that characterizes the factors enabling effective learning with PI, focusing primarily on π-Distill and characterizing when OPSD is competitive.
Training methods that utilize privileged information for language model distillation in multi-turn environments outperform standard supervised fine-tuning followed by reinforcement learning approaches.
privileged information reinforcement learning distillation teacher-student objective on-policy self-distillation reverse KL-penalty chain-of-thought supervision