LUT-LLM: Efficient Large Language Model Inference with Memory-based
  Computations on FPGAs
The rapid progress of large language models (LLMs) has advanced numerous
applications, yet efficient single-batch inference remains vital for on-device
intelligence. While FPGAs offer fine-grained data control and high energy
efficiency, recent GPU optimizations have narrowed their advantage, especially
under arithmetic-based computation. To overcome this, we leverage FPGAs'
abundant on-chip memory to shift LLM inference from arithmetic- to memory-based
computation through table lookups. We present LUT-LLM, the first FPGA
accelerator enabling 1B+ LLM inference via vector-quantized memory operations.
Our analysis identifies activation-weight co-quantization as the most effective
scheme, supported by (1) bandwidth-aware parallel centroid search, (2)
efficient 2D table lookups, and (3) a spatial-temporal hybrid design minimizing
data caching. Implemented on an AMD V80 FPGA for a customized Qwen 3 1.7B
model, LUT-LLM achieves 1.66x lower latency than AMD MI210 and 1.72x higher
energy efficiency than NVIDIA A100, scaling to 32B models with 2.16x efficiency
gain over A100.
6912be89a644ba07c499c7c7
Zifan He
claimed_verified
2026-04-02T08:18:53.764Z
6542bbc0bcc3731b0debc628
/avatars/7bcbff0738cf7efb9fbff73084446ea6.svg
Zifan He
OswaldHe123
user
OswaldHe123
6912be89a644ba07c499c7c8
Shengyu Ye
6912be89a644ba07c499c7c9
Rui Ma
6912be89a644ba07c499c7ca
Yang Wang
6912be89a644ba07c499c7cb
Jason Cong
LUT-LLM: Efficient Large Language Model Inference with Memory-based
  Computations on FPGAs
The rapid progress of large language models (LLMs) has advanced numerous
applications, yet efficient single-batch inference remains vital for on-device
intelligence. While FPGAs offer fine-grained data control and high energy
efficiency, recent GPU optimizations have narrowed their advantage, especially
under arithmetic-based computation. To overcome this, we leverage FPGAs'
abundant on-chip memory to shift LLM inference from arithmetic- to memory-based
computation through table lookups. We present LUT-LLM, the first FPGA
accelerator enabling 1B+ LLM inference via vector-quantized memory operations.
Our analysis identifies activation-weight co-quantization as the most effective
scheme, supported by (1) bandwidth-aware parallel centroid search, (2)
efficient 2D table lookups, and (3) a spatial-temporal hybrid design minimizing
data caching. Implemented on an AMD V80 FPGA for a customized Qwen 3 1.7B
model, LUT-LLM achieves 1.66x lower latency than AMD MI210 and 1.72x higher
energy efficiency than NVIDIA A100, scaling to 32B models with 2.16x efficiency
gain over A100.
LUT-LLM, an FPGA accelerator, improves LLM inference efficiency by shifting computation to memory-based operations, achieving lower latency and higher energy efficiency compared to GPUs.
LLMs FPGAs memory-based computation table lookups LUT-LLM vector-quantized memory operations activation-weight co-quantization bandwidth-aware parallel centroid search efficient 2D table lookups spatial-temporal hybrid design Qwen 3 1.7B AMD V80 FPGA AMD MI210 NVIDIA A100