Parallel Loop Transformer for Efficient Test-Time Computation Scaling
Large Language Models (LLMs) are powerful but often too slow and costly for
real-world use during inference. Looped transformers save on parameters by
reusing the same weights for multiple computational steps, or "loops." However,
this approach has a major flaw: the loops run one after another, causing
inference latency and memory requirements to increase with each added loop.
This makes them impractical for fast applications. To solve this problem, we
introduce the Parallel Loop Transformer (PLT). PLT is a new architecture that
delivers the performance benefits of a deep, looped model but with the low
latency of a standard, non-looped model. PLT works using two key techniques.
First, Cross-Loop Parallelism (CLP) breaks the sequential dependency by
computing different loops for different tokens at the same time, all within a
single pass. Second, to prevent memory costs from growing, we use an Efficient
Representation Enhancement strategy. This method shares the memory (KV cache)
from the first loop with all other loops. It then uses a Gated Sliding-Window
Attention (G-SWA) to combine this shared global information with local
information, maintaining high accuracy. Our experiments show that PLT achieves
the high accuracy of a traditional looped model but with almost no extra
latency or memory cost compared to a standard transformer.
6902c64272739622ee92a74f
Bohong Wu
6902c64272739622ee92a750
Mengzhao Chen
6902c64272739622ee92a751
Xiang Luo
6902c64272739622ee92a752
Shen Yan
6902c64272739622ee92a753
Qifan Yu
6902c64272739622ee92a754
Fan Xia
6902c64272739622ee92a755
Tianqi Zhang
6902c64272739622ee92a756
Hongrui Zhan
6902c64272739622ee92a757
Zheng Zhong
6902c64272739622ee92a758
Xun Zhou
6902c64272739622ee92a759
Siyuan Qiao
6902c64272739622ee92a75a
Xingyan Bin
Parallel Loop Transformer for Efficient Test-Time Computation Scaling
Large Language Models (LLMs) are powerful but often too slow and costly for
real-world use during inference. Looped transformers save on parameters by
reusing the same weights for multiple computational steps, or "loops." However,
this approach has a major flaw: the loops run one after another, causing
inference latency and memory requirements to increase with each added loop.
This makes them impractical for fast applications. To solve this problem, we
introduce the Parallel Loop Transformer (PLT). PLT is a new architecture that
delivers the performance benefits of a deep, looped model but with the low
latency of a standard, non-looped model. PLT works using two key techniques.
First, Cross-Loop Parallelism (CLP) breaks the sequential dependency by
computing different loops for different tokens at the same time, all within a
single pass. Second, to prevent memory costs from growing, we use an Efficient
Representation Enhancement strategy. This method shares the memory (KV cache)
from the first loop with all other loops. It then uses a Gated Sliding-Window
Attention (G-SWA) to combine this shared global information with local
information, maintaining high accuracy. Our experiments show that PLT achieves
the high accuracy of a traditional looped model but with almost no extra
latency or memory cost compared to a standard transformer.
The Parallel Loop Transformer (PLT) reduces inference latency and memory costs while maintaining high accuracy by using Cross-Loop Parallelism and Gated Sliding-Window Attention.
Parallel Loop Transformer PLT Cross-Loop Parallelism CLP Efficient Representation Enhancement Gated Sliding-Window Attention G-SWA