Scalable Prompt Routing via Fine-Grained Latent Task Discovery
Prompt routing dynamically selects the most appropriate large language model from a pool of candidates for each query, optimizing performance while managing costs. As model pools scale to include dozens of frontier models with narrow performance gaps, existing approaches face significant challenges: manually defined task taxonomies cannot capture fine-grained capability distinctions, while monolithic routers struggle to differentiate subtle differences across diverse tasks. We propose a two-stage routing architecture that addresses these limitations through automated fine-grained task discovery and task-aware quality estimation. Our first stage employs graph-based clustering to discover latent task types and trains a classifier to assign prompts to discovered tasks. The second stage uses a mixture-of-experts architecture with task-specific prediction heads for specialized quality estimates. At inference, we aggregate predictions from both stages to balance task-level stability with prompt-specific adaptability. Evaluated on 10 benchmarks with 11 frontier models, our method consistently outperforms existing baselines and surpasses the strongest individual model while incurring less than half its cost.
69c187fbe9374d7b3df94e2b
Yunyi Zhang
claimed_verified
2026-03-24T09:22:37.475Z
674253cfd7a6901ea8ce6567
/avatars/7770472571225eef5d4364a16378835c.svg
Yunyi Zhang
zhangyy114
user
zhangyy114
69c187fbe9374d7b3df94e2c
Soji Adeshina
69c187fbe9374d7b3df94e2d
Patrick Guan
69c187fbe9374d7b3df94e2e
Ashwin Ganesh
claimed_verified
2026-03-25T10:51:24.770Z
6938950544c833f480baa3c6
/avatars/fb82862de068cb12b5688c1ce78bb20b.svg
Ashwin Ganesh
ganesasp
user
ganesasp
69c187fbe9374d7b3df94e2f
Zhen Han
69c187fbe9374d7b3df94e30
Vassilis N. Ioannidis
69c187fbe9374d7b3df94e31
Huzefa Rangwala
69c187fbe9374d7b3df94e32
George Karypis
Scalable Prompt Routing via Fine-Grained Latent Task Discovery
Prompt routing dynamically selects the most appropriate large language model from a pool of candidates for each query, optimizing performance while managing costs. As model pools scale to include dozens of frontier models with narrow performance gaps, existing approaches face significant challenges: manually defined task taxonomies cannot capture fine-grained capability distinctions, while monolithic routers struggle to differentiate subtle differences across diverse tasks. We propose a two-stage routing architecture that addresses these limitations through automated fine-grained task discovery and task-aware quality estimation. Our first stage employs graph-based clustering to discover latent task types and trains a classifier to assign prompts to discovered tasks. The second stage uses a mixture-of-experts architecture with task-specific prediction heads for specialized quality estimates. At inference, we aggregate predictions from both stages to balance task-level stability with prompt-specific adaptability. Evaluated on 10 benchmarks with 11 frontier models, our method consistently outperforms existing baselines and surpasses the strongest individual model while incurring less than half its cost.
A two-stage prompt routing architecture uses graph-based clustering and mixture-of-experts learning to efficiently select optimal language models for queries, achieving superior performance at reduced computational cost.
prompt routing large language models task taxonomies graph-based clustering mixture-of-experts architecture task-specific prediction heads prompt classification task-aware quality estimation