DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput.
  We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines.
  Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87times on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96times without violating SLO.
69a0230eebfce7fbcca91ce5
Yongtong Wu
admin_assigned
2026-03-02T10:55:29.109Z
661d0758fc6943ea4b39851b
/avatars/37798f08e90f03c6e8c6e42f6955379a.svg
yongtong wu
jokerwyt
user
jokerwyt
69a0230eebfce7fbcca91ce6
Shaoyuan Chen
admin_assigned
2026-03-02T10:55:35.945Z
6657fb58a496f7fcb6b54c5d
/avatars/1e476f173b8be7dd622a951fab4b5bae.svg
Shaoyuan Chen
sy-chen
user
sy-chen
69a0230eebfce7fbcca91ce7
Yinmin Zhong
admin_assigned
2026-03-02T10:55:42.570Z
64e5d04767a759dd7155a445
/avatars/70090182b71db4de31fb8c325d25e73b.svg
Yinmin Zhong
PKUFlyingPig
user
PKUFlyingPig
69a0230eebfce7fbcca91ce8
Rilin Huang
admin_assigned
2026-03-02T10:55:57.466Z
67d299681f2a1c5c28c8601b
/avatars/87dab0a43f5285dd438b5b8da6722e68.svg
RILINHUANG
RILIN
user
RILIN
69a0230eebfce7fbcca91ce9
Yixuan Tan
69a0230eebfce7fbcca91cea
Wentao Zhang
69a0230eebfce7fbcca91ceb
Liyue Zhang
69a0230eebfce7fbcca91cec
Shangyan Zhou
admin_assigned
2026-03-02T10:56:55.470Z
654453e19b639f21e1d77d16
/avatars/079ec500c2ca7a31f6cb754b8c7ef065.svg
Shangyan Zhou
syzhou
user
syzhou
69a0230eebfce7fbcca91ced
Yuxuan Liu
69a0230eebfce7fbcca91cee
Shunfeng Zhou
69a0230eebfce7fbcca91cef
Mingxing Zhang
69a0230eebfce7fbcca91cf0
Xin Jin
69a0230eebfce7fbcca91cf1
Panpan Huang
DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput.
  We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines.
  Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87times on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96times without violating SLO.
DualPath addresses KV-cache storage I/O bottlenecks in multi-turn LLM inference by introducing dual-path loading and dynamic load balancing across prefill and decode engines.
KV-Cache prefill engines decoding engines RDMA global scheduler offline inference online serving