SkillFactory: Self-Distillation For Learning Cognitive Behaviors
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL; (2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
693133f52d1e5b0a7d84da69
Zayne Sprague
admin_assigned
2025-12-04T11:13:12.681Z
64e78a03e3953cd90bcad620
https://cdn-avatars.huggingface.co/v1/production/uploads/64e78a03e3953cd90bcad620/Rj_-xLJUsxdRmNhvRbssq.jpeg
Zayne Sprague
Zaynes
user
Zaynes
693133f52d1e5b0a7d84da6a
Jack Lu
693133f52d1e5b0a7d84da6b
Manya Wadhwa
claimed_verified
2025-12-16T22:33:07.662Z
655ab2ccc11dee7f7e6db119
/avatars/1c13e338cd4cc0b4eb681ed8f33abf19.svg
Manya Wadhwa
wadhma
user
wadhma
693133f52d1e5b0a7d84da6c
Sedrick Keh
admin_assigned
2025-12-04T11:12:53.518Z
6365c1158b6bd0d9f6605d3d
/avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
Sedrick Keh
sedrickkeh
user
sedrickkeh
693133f52d1e5b0a7d84da6d
Mengye Ren
693133f52d1e5b0a7d84da6e
Greg Durrett
admin_assigned
2025-12-04T11:13:18.166Z
65be9918b54ab5b37d1b67a7
/avatars/9953707affb6881724c8efb2abf0c668.svg
Greg Durrett
gregdurrett
user
gregdurrett
SkillFactory: Self-Distillation For Learning Cognitive Behaviors
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL; (2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
SkillFactory is a method for fine-tuning models to learn cognitive skills through supervised fine-tuning before reinforcement learning, enhancing their robustness and generalization post-RL.
reasoning models long chains of thought verification backtracking retrying reinforcement learning SkillFactory fine-tuning supervised fine-tuning silver SFT traces cognitive skills robustness generalization regression inductive biases