Batch Speculative Decoding Done Right
Speculative decoding speeds up LLM inference by using a small draft model to
propose multiple tokens that a target model verifies in parallel. Extending
this idea to batches is essential for production serving, but it introduces the
ragged tensor problem: sequences in the same batch accept different numbers of
draft tokens, breaking right-alignment and corrupting position IDs, attention
masks, and KV-cache state. We show that several existing batch implementations
violate output equivalence-the fundamental requirement that speculative
decoding must produce identical token sequences to standard autoregressive
generation. These violations occur precisely due to improper handling of the
ragged tensor problem. In response, we (1) characterize the synchronization
requirements that guarantee correctness, (2) present a correctness-first batch
speculative decoding EQSPEC that exposes realignment as consuming 40% of
overhead, and (3) introduce EXSPEC, which maintains a sliding pool of sequences
and dynamically forms same-length groups, to reduce the realignment overhead
while preserving per-sequence speculative speedups. On the SpecBench dataset,
across Vicuna-7B/68M, Qwen3-8B/0.6B, and GLM-4-9B/0.6B target/draft pairs, our
approach achieves up to 3times throughput improvement at batch size 8
compared to batch size 1, with efficient scaling through batch size 8, while
maintaining 95% output equivalence. Our method requires no custom kernels and
integrates cleanly with existing inference stacks. Our code is available at
https://github.com/eBay/spec_dec.
6900e9a5646208eac0d1f08a
Ranran Haoran Zhang
claimed_verified
2025-10-29T12:49:25.762Z
63c726b18bfd6a208eb5b125
/avatars/28a3b81d8a5b1a2af379e8dbeae799bd.svg
Ranran Haoran Zhang
windchimeran
user
windchimeran
6900e9a5646208eac0d1f08b
Soumik Dey
claimed_verified
2025-10-31T14:28:20.344Z
63f2cd22be95ed4c9a9f48f2
/avatars/18a0c0553ea067b18eb28f90a726953b.svg
Soumik Dey
zionsoumik
user
zionsoumik
6900e9a5646208eac0d1f08c
Ashirbad Mishra
6900e9a5646208eac0d1f08d
Hansi Wu
6900e9a5646208eac0d1f08e
Binbin Li
6900e9a5646208eac0d1f08f
Rui Zhang
Batch Speculative Decoding Done Right
Speculative decoding speeds up LLM inference by using a small draft model to
propose multiple tokens that a target model verifies in parallel. Extending
this idea to batches is essential for production serving, but it introduces the
ragged tensor problem: sequences in the same batch accept different numbers of
draft tokens, breaking right-alignment and corrupting position IDs, attention
masks, and KV-cache state. We show that several existing batch implementations
violate output equivalence-the fundamental requirement that speculative
decoding must produce identical token sequences to standard autoregressive
generation. These violations occur precisely due to improper handling of the
ragged tensor problem. In response, we (1) characterize the synchronization
requirements that guarantee correctness, (2) present a correctness-first batch
speculative decoding EQSPEC that exposes realignment as consuming 40% of
overhead, and (3) introduce EXSPEC, which maintains a sliding pool of sequences
and dynamically forms same-length groups, to reduce the realignment overhead
while preserving per-sequence speculative speedups. On the SpecBench dataset,
across Vicuna-7B/68M, Qwen3-8B/0.6B, and GLM-4-9B/0.6B target/draft pairs, our
approach achieves up to 3times throughput improvement at batch size 8
compared to batch size 1, with efficient scaling through batch size 8, while
maintaining 95% output equivalence. Our method requires no custom kernels and
integrates cleanly with existing inference stacks. Our code is available at
https://github.com/eBay/spec_dec.
Batch speculative decoding improves LLM inference throughput by managing ragged tensors to maintain output equivalence and reduce realignment overhead.
speculative decoding draft model target model ragged tensor problem position IDs attention masks KV-cache state output equivalence autoregressive generation synchronization requirements EQSPEC EXSPEC SpecBench Vicuna-7B/68M Qwen3-8B/0.6B GLM-4-9B/0.6B