RelayGen: Intra-Generation Model Switching for Efficient Reasoning
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks by generating long, multi-step reasoning trajectories, but inference-time scaling incurs substantial deployment cost. A key challenge is that generation difficulty varies within a single output, whereas existing efficiency-oriented approaches either ignore this intra-generation variation or rely on supervised token-level routing with high system complexity. We present RelayGen, a training-free, segment-level runtime model switching framework that exploits difficulty variation in long-form reasoning. Through offline analysis of generation uncertainty using token probability margins, we show that coarse-grained segment-level control is sufficient to capture difficulty transitions within a reasoning trajectory. RelayGen identifies model-specific switch cues that signal transitions to lower-difficulty segments and dynamically delegates their continuation to a smaller model, while preserving high-difficulty reasoning on the large model. Across multiple reasoning benchmarks, RelayGen substantially reduces inference latency while preserving most of the accuracy of large models. When combined with speculative decoding, RelayGen achieves up to 2.2times end-to-end speedup with less than 2\% accuracy degradation, without requiring additional training or learned routing components.
698a9d531b2dc6b37d61af3d
Jiwon Song
claimed_verified
2026-02-10T09:06:44.201Z
662672eaebdfec5cfdf1d034
https://cdn-avatars.huggingface.co/v1/production/uploads/662672eaebdfec5cfdf1d034/RhsKly3KvbtPkDuVnEdWb.jpeg
Jiwon Song
jiwonsong
user
jiwonsong
698a9d531b2dc6b37d61af3e
Yoongon Kim
698a9d531b2dc6b37d61af3f
Jae-Joon Kim
RelayGen: Intra-Generation Model Switching for Efficient Reasoning
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks by generating long, multi-step reasoning trajectories, but inference-time scaling incurs substantial deployment cost. A key challenge is that generation difficulty varies within a single output, whereas existing efficiency-oriented approaches either ignore this intra-generation variation or rely on supervised token-level routing with high system complexity. We present RelayGen, a training-free, segment-level runtime model switching framework that exploits difficulty variation in long-form reasoning. Through offline analysis of generation uncertainty using token probability margins, we show that coarse-grained segment-level control is sufficient to capture difficulty transitions within a reasoning trajectory. RelayGen identifies model-specific switch cues that signal transitions to lower-difficulty segments and dynamically delegates their continuation to a smaller model, while preserving high-difficulty reasoning on the large model. Across multiple reasoning benchmarks, RelayGen substantially reduces inference latency while preserving most of the accuracy of large models. When combined with speculative decoding, RelayGen achieves up to 2.2times end-to-end speedup with less than 2\% accuracy degradation, without requiring additional training or learned routing components.
RelayGen is a training-free framework that dynamically switches between large and small reasoning models during inference based on segment-level difficulty estimation, achieving faster execution with minimal accuracy loss.
large reasoning models multi-step reasoning trajectories inference-time scaling token probability margins segment-level control model switching speculative decoding