Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to sim2times at matched accuracy.
693919e9dfc35938ba12a151
Monishwaran Maheswaran
claimed_verified
2025-12-11T10:13:45.903Z
647f063c4da748fc4abb2a13
/avatars/5885df03911cbd25493143eaee75475a.svg
Monishwaran
Nietzsche6700
user
Nietzsche6700
693919e9dfc35938ba12a152
Rishabh Tiwari
693919e9dfc35938ba12a153
Yuezhou Hu
claimed_verified
2025-12-10T08:39:25.344Z
67dc66fe55c24fc4f981a4ab
/avatars/7bd900ade802d99db7c562ad6c2f6661.svg
Yuezhou Hu
yuezhouhu
user
yuezhouhu
693919e9dfc35938ba12a154
Kerem Dilmen
693919e9dfc35938ba12a155
Coleman Hooper
693919e9dfc35938ba12a156
Haocheng Xi
693919e9dfc35938ba12a157
Nicholas Lee
693919e9dfc35938ba12a158
Mehrdad Farajtabar
693919e9dfc35938ba12a159
Michael W. Mahoney
693919e9dfc35938ba12a15a
Kurt Keutzer
693919e9dfc35938ba12a15b
Amir Gholami
Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to sim2times at matched accuracy.
Arbitrage is a dynamic routing framework for speculative decoding that improves efficiency in large language model inference by predicting when the target model will provide a better step, outperforming traditional step-level methods.
Speculative Decoding draft model target model token-level Speculative Decoding step-level semantic verification Arbitrage router Arbitrage Oracle inference latency