RelayLLM: Efficient Reasoning via Collaborative Decoding
Large Language Models (LLMs) for complex reasoning is often hindered by high computational costs and latency, while resource-efficient Small Language Models (SLMs) typically lack the necessary reasoning capacity. Existing collaborative approaches, such as cascading or routing, operate at a coarse granularity by offloading entire queries to LLMs, resulting in significant computational waste when the SLM is capable of handling the majority of reasoning steps. To address this, we propose RelayLLM, a novel framework for efficient reasoning via token-level collaborative decoding. Unlike routers, RelayLLM empowers the SLM to act as an active controller that dynamically invokes the LLM only for critical tokens via a special command, effectively "relaying" the generation process. We introduce a two-stage training framework, including warm-up and Group Relative Policy Optimization (GRPO) to teach the model to balance independence with strategic help-seeking. Empirical results across six benchmarks demonstrate that RelayLLM achieves an average accuracy of 49.52%, effectively bridging the performance gap between the two models. Notably, this is achieved by invoking the LLM for only 1.07% of the total generated tokens, offering a 98.2% cost reduction compared to performance-matched random routers.
69606c325b7998385e639481
Chengsong Huang
claimed_verified
2026-01-09T08:35:21.851Z
62ea79dd01ed9b0e8f61ccd3
/avatars/70af83e0e267be39fcd5f23b85e2dafa.svg
Chengsong Huang
ChengsongHuang
user
ChengsongHuang
69606c325b7998385e639482
Tong Zheng
69606c325b7998385e639483
Langlin Huang
admin_assigned
2026-01-09T15:53:24.796Z
65e02d89574e5aa0e9ce3efa
/avatars/2ab152a10b21d81fb1defc726b8e951a.svg
Langlin Huang
shrango
user
shrango
69606c325b7998385e639484
Jinyuan Li
claimed_verified
2026-02-05T10:57:41.084Z
64fc20d899123d7698a30e61
/avatars/9231982cf70a0689f50accedf1004702.svg
Jinyuan Li
jinyuan222
user
jinyuan222
69606c325b7998385e639485
Haolin Liu
69606c325b7998385e639486
Jiaxin Huang
RelayLLM: Efficient Reasoning via Collaborative Decoding
Large Language Models (LLMs) for complex reasoning is often hindered by high computational costs and latency, while resource-efficient Small Language Models (SLMs) typically lack the necessary reasoning capacity. Existing collaborative approaches, such as cascading or routing, operate at a coarse granularity by offloading entire queries to LLMs, resulting in significant computational waste when the SLM is capable of handling the majority of reasoning steps. To address this, we propose RelayLLM, a novel framework for efficient reasoning via token-level collaborative decoding. Unlike routers, RelayLLM empowers the SLM to act as an active controller that dynamically invokes the LLM only for critical tokens via a special command, effectively "relaying" the generation process. We introduce a two-stage training framework, including warm-up and Group Relative Policy Optimization (GRPO) to teach the model to balance independence with strategic help-seeking. Empirical results across six benchmarks demonstrate that RelayLLM achieves an average accuracy of 49.52%, effectively bridging the performance gap between the two models. Notably, this is achieved by invoking the LLM for only 1.07% of the total generated tokens, offering a 98.2% cost reduction compared to performance-matched random routers.
RelayLLM enables efficient collaborative reasoning between small and large language models through token-level dynamic invocation, achieving high accuracy with minimal computational overhead.
Large Language Models Small Language Models collaborative decoding token-level collaboration Group Relative Policy Optimization policy optimization dynamic invocation computational efficiency reasoning capacity