GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts
Large Reasoning Models (LRMs) achieve remarkable performance by explicitly generating multi-step chains of thought, but this capability incurs substantial inference latency and computational cost. Collaborative inference offers a promising solution by selectively allocating work between lightweight and large models, yet a fundamental challenge remains: determining when a reasoning step requires the capacity of a large model or the efficiency of a small model. Existing routing strategies either rely on local token probabilities or post-hoc verification, introducing significant inference overhead. In this work, we propose a novel perspective on step-wise collaboration: the difficulty of a reasoning step can be inferred from its very first token. Inspired by the "Aha Moment" phenomenon in LRMs, we show that the entropy of the initial token serves as a strong predictor of step difficulty. Building on this insight, we introduce GlimpRouter, a training-free step-wise collaboration framework. GlimpRouter employs a lightweight model to generate only the first token of each reasoning step and routes the step to a larger model only when the initial token entropy exceeds a threshold. Experiments on multiple benchmarks demonstrate that our approach significantly reduces inference latency while preserving accuracy. For instance, GlimpRouter attains a substantial 10.7% improvement in accuracy while reducing inference latency by 25.9% compared to a standalone large model on AIME25. These results suggest a simple yet effective mechanism for reasoning: allocating computation based on a glimpse of thought rather than full-step evaluation.
696493ff138cc47cbd7653ab
Wenhao Zeng
696493ff138cc47cbd7653ac
Xuteng Zhang
696493ff138cc47cbd7653ad
Yuling Shi
claimed_verified
2026-01-12T10:33:29.307Z
645b0c3ec35da9c7afd95421
https://cdn-avatars.huggingface.co/v1/production/uploads/645b0c3ec35da9c7afd95421/vYBrCDagHsXAo6J2p-uG0.jpeg
Yuling
YerbaPage
user
YerbaPage
696493ff138cc47cbd7653ae
Chao Hu
696493ff138cc47cbd7653af
Yuting Chen
696493ff138cc47cbd7653b0
Beijun Shen
696493ff138cc47cbd7653b1
Xiaodong Gu
claimed_verified
2026-01-28T11:32:23.214Z
6624ede5478dada13051af60
/avatars/1b064f7e3c417f1b603909ff3dbf2430.svg
xiaodong gu
guxiaodong1987
user
guxiaodong1987
GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts
Large Reasoning Models (LRMs) achieve remarkable performance by explicitly generating multi-step chains of thought, but this capability incurs substantial inference latency and computational cost. Collaborative inference offers a promising solution by selectively allocating work between lightweight and large models, yet a fundamental challenge remains: determining when a reasoning step requires the capacity of a large model or the efficiency of a small model. Existing routing strategies either rely on local token probabilities or post-hoc verification, introducing significant inference overhead. In this work, we propose a novel perspective on step-wise collaboration: the difficulty of a reasoning step can be inferred from its very first token. Inspired by the "Aha Moment" phenomenon in LRMs, we show that the entropy of the initial token serves as a strong predictor of step difficulty. Building on this insight, we introduce GlimpRouter, a training-free step-wise collaboration framework. GlimpRouter employs a lightweight model to generate only the first token of each reasoning step and routes the step to a larger model only when the initial token entropy exceeds a threshold. Experiments on multiple benchmarks demonstrate that our approach significantly reduces inference latency while preserving accuracy. For instance, GlimpRouter attains a substantial 10.7% improvement in accuracy while reducing inference latency by 25.9% compared to a standalone large model on AIME25. These results suggest a simple yet effective mechanism for reasoning: allocating computation based on a glimpse of thought rather than full-step evaluation.
Large reasoning models' inference latency can be reduced by routing reasoning steps to larger models based on the entropy of their first token, enabling efficient collaborative inference without additional training.
large reasoning models multi-step chains of thought inference latency computational cost collaborative inference routing strategies token probabilities post-hoc verification step-wise collaboration reasoning step difficulty initial token entropy Aha Moment phenomenon training-free framework lightweight model large model threshold routing entropy-based prediction inference overhead GlimpRouter