THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.
69801ae26676f9332270659b
Seanie Lee
claimed_verified
2026-02-02T16:52:56.623Z
64ad5f59b7e4b2c1ce47eb43
/avatars/1f13ebe21a90d8c99920aa2c8cd9ac45.svg
Seanie Lee
Seanie-lee
user
Seanie-lee
69801ae26676f9332270659c
Sangwoo Park
claimed_verified
2026-02-02T16:52:59.260Z
638716c14e00d7fc0902fef4
/avatars/5fa8152f8c0e4e600d1a64802c3e0103.svg
Sangwoo Park
Sangsang
user
Sangsang
69801ae26676f9332270659d
Yumin Choi
claimed_verified
2026-02-02T16:52:54.423Z
64cfa0b9749587dbe01d0079
/avatars/93ca0a1d9c5578d052c5af0d4d1a0252.svg
Yumin Choi
YuminChoi
user
YuminChoi
69801ae26676f9332270659e
Gyeongman Kim
69801ae26676f9332270659f
Minki Kang
69801ae26676f933227065a0
Jihun Yun
69801ae26676f933227065a1
Dongmin Park
69801ae26676f933227065a2
Jongho Park
69801ae26676f933227065a3
Sung Ju Hwang
THINKSAFE: Self-Generated Safety Alignment for Reasoning Models
Large reasoning models (LRMs) achieve remarkable performance by leveraging reinforcement learning (RL) on reasoning tasks to generate long chain-of-thought (CoT) reasoning. However, this over-optimization often prioritizes compliance, making models vulnerable to harmful prompts. To mitigate this safety degradation, recent approaches rely on external teacher distillation, yet this introduces a distributional discrepancy that degrades native reasoning. We propose ThinkSafe, a self-generated alignment framework that restores safety alignment without external teachers. Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm. ThinkSafe unlocks this via lightweight refusal steering, guiding the model to generate in-distribution safety reasoning traces. Fine-tuning on these self-generated responses effectively realigns the model while minimizing distribution shift. Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency. Notably, it achieves superior safety and comparable reasoning to GRPO, with significantly reduced computational cost. Code, models, and datasets are available at https://github.com/seanie12/ThinkSafe.git.
ThinkSafe is a self-aligned framework that enhances safety in large reasoning models through lightweight refusal steering and fine-tuning on self-generated responses, maintaining reasoning performance while reducing computational costs.
reinforcement learning chain-of-thought reasoning external teacher distillation distributional discrepancy lightweight refusal steering self-generated alignment safety alignment reasoning proficiency computational cost