Token-Level LLM Collaboration via FusionRoute
Large language models (LLMs) exhibit strengths across diverse domains. However, achieving strong performance across these domains with a single general-purpose model typically requires scaling to sizes that are prohibitively expensive to train and deploy. On the other hand, while smaller domain-specialized models are much more efficient, they struggle to generalize beyond their training distributions. To address this dilemma, we propose FusionRoute, a robust and effective token-level multi-LLM collaboration framework in which a lightweight router simultaneously (i) selects the most suitable expert at each decoding step and (ii) contributes a complementary logit that refines or corrects the selected expert's next-token distribution via logit addition. Unlike existing token-level collaboration methods that rely solely on fixed expert outputs, we provide a theoretical analysis showing that pure expert-only routing is fundamentally limited: unless strong global coverage assumptions hold, it cannot in general realize the optimal decoding policy. By augmenting expert selection with a trainable complementary generator, FusionRoute expands the effective policy class and enables recovery of optimal value functions under mild conditions. Empirically, across both Llama-3 and Gemma-2 families and diverse benchmarks spanning mathematical reasoning, code generation, and instruction following, FusionRoute outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning, while remaining competitive with domain experts on their respective tasks.
696073d35b7998385e6394c4
Nuoya Xiong
696073d35b7998385e6394c5
Yuhang Zhou
claimed_verified
2026-01-09T16:50:49.484Z
64887eb15cf73a16e767b56a
/avatars/ada2b6a07346b1d61322ddd04d219318.svg
Yuhang Zhou
zyhang1998
user
zyhang1998
696073d35b7998385e6394c6
Hanqing Zeng
claimed_verified
2026-01-12T10:36:17.428Z
6729196729fdd2b3fc360e21
/avatars/407f0660c8b2b63bf2b60aa9553bb2c6.svg
Hanqing Zeng
zimplex
user
zimplex
696073d35b7998385e6394c7
Zhaorun Chen
696073d35b7998385e6394c8
Furong Huang
696073d35b7998385e6394c9
Shuchao Bi
696073d35b7998385e6394ca
Lizhu Zhang
696073d35b7998385e6394cb
Zhuokai Zhao
claimed_verified
2026-01-12T10:36:19.518Z
63af25605fe9db73f67a0fb7
https://cdn-avatars.huggingface.co/v1/production/uploads/63af25605fe9db73f67a0fb7/mcQHbTdJVxi2GdHzRB9ad.jpeg
Zhuokai Zhao
zhuokai
user
zhuokai
Token-Level LLM Collaboration via FusionRoute
Large language models (LLMs) exhibit strengths across diverse domains. However, achieving strong performance across these domains with a single general-purpose model typically requires scaling to sizes that are prohibitively expensive to train and deploy. On the other hand, while smaller domain-specialized models are much more efficient, they struggle to generalize beyond their training distributions. To address this dilemma, we propose FusionRoute, a robust and effective token-level multi-LLM collaboration framework in which a lightweight router simultaneously (i) selects the most suitable expert at each decoding step and (ii) contributes a complementary logit that refines or corrects the selected expert's next-token distribution via logit addition. Unlike existing token-level collaboration methods that rely solely on fixed expert outputs, we provide a theoretical analysis showing that pure expert-only routing is fundamentally limited: unless strong global coverage assumptions hold, it cannot in general realize the optimal decoding policy. By augmenting expert selection with a trainable complementary generator, FusionRoute expands the effective policy class and enables recovery of optimal value functions under mild conditions. Empirically, across both Llama-3 and Gemma-2 families and diverse benchmarks spanning mathematical reasoning, code generation, and instruction following, FusionRoute outperforms both sequence- and token-level collaboration, model merging, and direct fine-tuning, while remaining competitive with domain experts on their respective tasks.
FusionRoute is a token-level multi-LLM collaboration framework that uses a lightweight router to select optimal experts and add complementary logits, outperforming existing methods in diverse tasks while maintaining efficiency.
large language models token-level collaboration multi-LLM collaboration lightweight router expert selection logit addition complementary generator optimal decoding policy model merging direct fine-tuning