RDMA Point-to-Point Communication for LLM Systems
Emerging Large Language Model (LLM) system patterns, such as disaggregated
inference, Mixture-of-Experts (MoE) routing, and asynchronous reinforcement
fine-tuning, require flexible point-to-point communication beyond simple
collectives. Existing implementations are locked to specific Network Interface
Controllers (NICs), hindering integration into inference engines and
portability across hardware providers. We present TransferEngine, which bridges
the functionality of common NICs to expose a uniform interface. TransferEngine
exposes one-sided WriteImm operations with a ImmCounter primitive for
completion notification, without ordering assumptions of network transport,
transparently managing multiple NICs per GPU. We demonstrate peak throughput of
400 Gbps on both NVIDIA ConnectX-7 and AWS Elastic Fabric Adapter (EFA). We
showcase TransferEngine through three production systems: (1) KvCache transfer
for disaggregated inference with dynamic scaling, (2) RL weight updates
achieving 1.3 seconds for trillion-parameter models, and (3) MoE
dispatch/combine implementation exceeding DeepEP decode latency on ConnectX-7,
with the first viable latencies on EFA. We demonstrate that our portable
point-to-point communication complements collectives while avoiding lock-in.
6908364f812eca10f9cc5edc
Nandor Licker
6908364f812eca10f9cc5edd
Kevin Hu
6908364f812eca10f9cc5ede
Vladimir Zaytsev
6908364f812eca10f9cc5edf
Lequn Chen
claimed_verified
2026-03-30T09:12:54.587Z
63f6a8f9e94ed998c460302a
/avatars/c2b78e2725a5c8c1cd3ab2e502afff71.svg
Lequn Chen
abcdabcd987
user
abcdabcd987
RDMA Point-to-Point Communication for LLM Systems
Emerging Large Language Model (LLM) system patterns, such as disaggregated
inference, Mixture-of-Experts (MoE) routing, and asynchronous reinforcement
fine-tuning, require flexible point-to-point communication beyond simple
collectives. Existing implementations are locked to specific Network Interface
Controllers (NICs), hindering integration into inference engines and
portability across hardware providers. We present TransferEngine, which bridges
the functionality of common NICs to expose a uniform interface. TransferEngine
exposes one-sided WriteImm operations with a ImmCounter primitive for
completion notification, without ordering assumptions of network transport,
transparently managing multiple NICs per GPU. We demonstrate peak throughput of
400 Gbps on both NVIDIA ConnectX-7 and AWS Elastic Fabric Adapter (EFA). We
showcase TransferEngine through three production systems: (1) KvCache transfer
for disaggregated inference with dynamic scaling, (2) RL weight updates
achieving 1.3 seconds for trillion-parameter models, and (3) MoE
dispatch/combine implementation exceeding DeepEP decode latency on ConnectX-7,
with the first viable latencies on EFA. We demonstrate that our portable
point-to-point communication complements collectives while avoiding lock-in.
TransferEngine provides a uniform interface for flexible point-to-point communication in large language models, supporting disaggregated inference, reinforcement learning, and Mixture-of-Experts routing across different hardware.
disaggregated inference Mixture-of-Experts (MoE) asynchronous reinforcement fine-tuning Network Interface Controllers (NICs) TransferEngine WriteImm operations ImmCounter peak throughput NVIDIA ConnectX-7 AWS Elastic Fabric Adapter (EFA) KvCache transfer RL weight updates MoE dispatch/combine DeepEP decode latency