The Transformers modeling backend is now as fast as native vLLM
vLLM PR #47187 (merged, @ 5bce653)
Run any Hugging Face model in vLLM via --model-impl transformers and get the throughput of a hand-written native implementation — from a 4B dense model to a 235B-parameter FP8 MoE. Measured on 8×H100.
Results (% of native vLLM)
| Model | Config | Before PR | After PR | vs native |
|---|---|---|---|---|
| Qwen3-4B | Dense · bf16 · TP1 | 98.7% | 100.0% | parity |
| Qwen3-32B | Dense · bf16 · TP2 | 98.4% | 100.1% | parity |
| Qwen3-235B-A22B-FP8 | MoE · FP8 · DP8+EP | 96.6% | 102.0% | +2.0% |
Each model is compared under three code paths on identical hardware/workload: native (--model-impl vllm, the bar to match), after (--model-impl transformers with the PR), and before (the same backend on the PR's parent commit).
Why the 235B MoE gains most
The PR routes Hugging Face MoE blocks through vLLM's FusedMoE — a grouped all-to-all expert dispatch the pre-PR naive per-expert loop can't match. At DP8 + expert parallel (the throughput-optimal MoE config) that's +5.6% over the pre-PR path, and still native parity. The QKV fusion also fixes a TP8+EP load failure — pre-PR the unfused k_proj shards to 64, not a multiple of the FP8 quant block (128).
Reproduce
Workload: random 1024-in / 128-out, 1000 prompts, torch.compile on, 8×H100. Point VLLM/VENV at your vLLM checkout (on PR #47187) and its venv, then ./benchmark.sh (or ./benchmark.sh q3_235b for one model).
Runner
benchmark.sh
#!/usr/bin/env bash
# =============================================================================
# Reproduce the benchmark behind vLLM PR #47187
# "Make the Transformers modeling backend as fast as native vLLM"
# =============================================================================
#
# What this measures
# ------------------
# For each model we compare THREE conditions on identical hardware, workload
# and flags — the only thing that changes is the code path:
#
# native --model-impl vllm vLLM's hand-written model (the bar to match)
# after --model-impl transformers generic Transformers backend, WITH PR #47187
# before --model-impl transformers generic Transformers backend, WITHOUT PR #47187
#
# PR #47187 is 100% Python and lives entirely under
# vllm/model_executor/models/transformers/ — no CUDA/C++/build changes. So the
# honest before/after delta is obtained by `git checkout`-ing just those files
# to the PR's parent commit and back. No recompile, nothing else moves.
#
# Every number the graphic reports is a WITHIN-model, WITHIN-harness ratio
# (% of native, and after/before), so the comparison is apples-to-apples even
# though absolute tok/s differ across model sizes and harnesses.
#
# Hardware used for the published figures: 8x NVIDIA H100 80GB.
#
# Usage
# -----
# ./benchmark.sh # run every condition for every model
# ./benchmark.sh q3_235b # run only conditions whose tag matches this
# Results (JSON + logs) land in ./results/. Re-running skips completed configs.
# =============================================================================
set -uo pipefail
# --- configure these two paths for your machine ------------------------------
VLLM=${VLLM:-$HOME/vllm} # editable vLLM checkout on PR #47187
VENV=${VENV:-$HOME/.venv/bin} # venv with that vLLM installed
# -----------------------------------------------------------------------------
PR=5bce653e09 # PR #47187 merge commit
PARENT="${PR}^" # its parent = "before"
TFDIR=vllm/model_executor/models/transformers/
OUT="$(cd "$(dirname "$0")" && pwd)/results"
PORT=8231
FILTER="${1:-}" # optional tag substring filter
mkdir -p "$OUT"
# Shared workload for every run: 1024-token prompts, 128-token generations.
INLEN=1024; OUTLEN=128; NPROMPTS=1000
# Model matrix that produced the graphic.
# tag | model | harness | tp | gpus | extra-flags
# harness = "throughput" (offline) or "serve" (online; required for data parallel).
MODELS=(
"q3_4b|Qwen/Qwen3-4B|throughput|1|0|"
"q3_32b|Qwen/Qwen3-32B|throughput|2|0,1|"
"q3_235b|Qwen/Qwen3-235B-A22B-FP8|serve|8|0,1,2,3,4,5,6,7|--data-parallel-size 8 --enable-expert-parallel --max-model-len 8192"
)
# Always leave the tree back on the PR files, whatever happens.
restore(){ git -C "$VLLM" checkout "$PR" -- "$TFDIR" 2>/dev/null; }
trap restore EXIT
# Node-local compile caches on tmpfs. On a networked filesystem the multi-rank
# inductor/triton autotune cache races and throws "CUDA driver error: file not
# found"; keeping caches on /dev/shm with single-threaded compile avoids it.
setcache(){
local tag=$1 c=/dev/shm/benchcache/$1
rm -rf "$c"; mkdir -p "$c"/{triton,inductor,vllm}
export TRITON_CACHE_DIR="$c/triton" TORCHINDUCTOR_CACHE_DIR="$c/inductor"
export VLLM_CACHE_ROOT="$c/vllm" TORCHINDUCTOR_COMPILE_THREADS=1
}
free_gpus(){ # wait until the GPUs are actually released before the next run
pkill -9 -f "EngineCore" 2>/dev/null; pkill -9 -f "vllm serve" 2>/dev/null
pkill -9 -f "VLLM::" 2>/dev/null
for _ in $(seq 1 90); do
local u; u=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits \
| awk '{s+=$1} END{print s+0}')
[ "$u" -lt 3000 ] && break; sleep 2
done
}
# ---- offline throughput harness (dense models) ------------------------------
run_throughput(){ # tag model impl tp gpus extra...
local tag=$1 model=$2 impl=$3 tp=$4 gpus=$5; shift 5
local json="$OUT/$tag.json" log="$OUT/$tag.log"
grep -q tokens_per_second "$json" 2>/dev/null && { echo "SKIP $tag (done)"; return; }
setcache "$tag"
echo ">>> $tag : throughput impl=$impl tp=$tp gpus=$gpus"
CUDA_VISIBLE_DEVICES=$gpus VLLM_LOGGING_LEVEL=INFO timeout 5400 \
"$VENV/vllm" bench throughput \
--model "$model" --model-impl "$impl" --tensor-parallel-size "$tp" \
--dataset-name random --input-len $INLEN --output-len $OUTLEN \
--num-prompts $NPROMPTS --output-json "$json" "$@" > "$log" 2>&1
echo -n " "; grep -o '"tokens_per_second":[^,]*' "$json" 2>/dev/null || echo "NO RESULT"
echo " fused-op log lines: $(grep -cE '^Fused:' "$log")"
free_gpus
}
# ---- online serving harness (data-parallel MoE) -----------------------------
run_serve(){ # tag model impl tp gpus extra...
local tag=$1 model=$2 impl=$3 tp=$4 gpus=$5; shift 5
local json="$OUT/$tag.json" slog="$OUT/${tag}_server.log" clog="$OUT/${tag}_client.log"
grep -q total_token_throughput "$json" 2>/dev/null && { echo "SKIP $tag (done)"; return; }
setcache "$tag"; rm -f "$json"
echo ">>> $tag : serve impl=$impl gpus=$gpus"
CUDA_VISIBLE_DEVICES=$gpus VLLM_LOGGING_LEVEL=INFO \
"$VENV/vllm" serve "$model" --model-impl "$impl" --port $PORT "$@" > "$slog" 2>&1 &
local spid=$! ready=0
for _ in $(seq 1 400); do
[ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$PORT/health)" = 200 ] \
&& { ready=1; break; }
kill -0 $spid 2>/dev/null || { echo " server died (see $slog)"; break; }
sleep 3
done
if [ "$ready" = 1 ]; then
echo " server ready -> client"
"$VENV/vllm" bench serve --model "$model" --host 127.0.0.1 --port $PORT \
--dataset-name random --random-input-len $INLEN --random-output-len $OUTLEN \
--num-prompts $NPROMPTS --ignore-eos --save-result --result-filename "$json" > "$clog" 2>&1
echo -n " "; grep -oE '"total_token_throughput":[^,}]*' "$json" 2>/dev/null || echo "NO RESULT"
echo " fused-op log lines: $(grep -cE '^Fused:' "$slog")"
fi
kill -9 $spid 2>/dev/null; free_gpus
}
dispatch(){ # spec impl condtag
IFS='|' read -r tag model harness tp gpus extra <<< "$1"
[ -n "$FILTER" ] && [[ "$tag" != *"$FILTER"* ]] && return
# shellcheck disable=SC2086
run_"$harness" "${tag}_$3" "$model" "$2" "$tp" "$gpus" $extra
}
# === Phase A: native + after — run on the PR files ===========================
echo "######## PHASE A: native + after (PR #47187 files) ########"
git -C "$VLLM" checkout "$PR" -- "$TFDIR"
for spec in "${MODELS[@]}"; do
dispatch "$spec" vllm native
dispatch "$spec" transformers after
done
# === Phase B: before — swap ONLY the transformers/ dir to the parent commit ==
echo "######## PHASE B: before (parent commit ${PARENT}) ########"
git -C "$VLLM" checkout "$PARENT" -- "$TFDIR"
git -C "$VLLM" status --porcelain "$TFDIR"
for spec in "${MODELS[@]}"; do
dispatch "$spec" transformers before
done
restore
echo "######## DONE — results in $OUT ; tree restored to PR files ########"
Raw results (vllm bench output)
q3_4b_native.json
{
"elapsed_time": 24.27075677085668,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 41.201846709648486,
"tokens_per_second": 47464.52740951506
}
q3_4b_after.json
{
"elapsed_time": 24.28168404311873,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 41.18330500570834,
"tokens_per_second": 47443.16736657601
}
q3_4b_before.json
{
"elapsed_time": 24.58867740095593,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 40.6691252113106,
"tokens_per_second": 46850.83224342981
}
q3_32b_native.json
{
"elapsed_time": 79.22332654520869,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 12.622544944882506,
"tokens_per_second": 14541.171776504647
}
q3_32b_after.json
{
"elapsed_time": 79.12049886095338,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 12.63894963247645,
"tokens_per_second": 14560.06997661287
}
q3_32b_before.json
{
"elapsed_time": 80.50363391195424,
"num_requests": 1000,
"total_num_tokens": 1152000,
"requests_per_second": 12.421799506512796,
"tokens_per_second": 14309.913031502741
}
q3_235b_native.json
{
"date": "20260707-115114",
"endpoint_type": "openai",
"backend": "openai",
"label": null,
"model_id": "Qwen/Qwen3-235B-A22B-FP8",
"tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
"num_prompts": 1000,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": null,
"duration": 35.44470150419511,
"completed": 1000,
"failed": 0,
"total_input_tokens": 1024000,
"total_output_tokens": 128000,
"request_throughput": 28.212961530558907,
"request_goodput": null,
"output_throughput": 3611.25907591154,
"total_token_throughput": 32501.331683203858,
"max_output_tokens_per_s": 16680.0,
"max_concurrent_requests": 1000,
"rtfx": 0.0,
"mean_ttft_ms": 15171.714233175619,
"median_ttft_ms": 13715.512390015647,
"std_ttft_ms": 9169.5805137529,
"p99_ttft_ms": 30926.241073654965,
"mean_tpot_ms": 116.817842786138,
"median_tpot_ms": 138.0034470041539,
"std_tpot_ms": 40.08971847574921,
"p99_tpot_ms": 160.9097329545656,
"mean_itl_ms": 116.81968330958155,
"median_itl_ms": 41.13172448705882,
"std_itl_ms": 319.89249689586126,
"p99_itl_ms": 1439.9876020080408
}
q3_235b_after.json
{
"date": "20260707-120324",
"endpoint_type": "openai",
"backend": "openai",
"label": null,
"model_id": "Qwen/Qwen3-235B-A22B-FP8",
"tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
"num_prompts": 1000,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": null,
"duration": 34.749073383864015,
"completed": 1000,
"failed": 0,
"total_input_tokens": 1024000,
"total_output_tokens": 128000,
"request_throughput": 28.77774578197407,
"request_goodput": null,
"output_throughput": 3683.551460092681,
"total_token_throughput": 33151.96314083412,
"max_output_tokens_per_s": 16526.0,
"max_concurrent_requests": 1000,
"rtfx": 0.0,
"mean_ttft_ms": 14851.332848654361,
"median_ttft_ms": 12809.239127556793,
"std_ttft_ms": 9119.170769998696,
"p99_ttft_ms": 30412.630736478604,
"mean_tpot_ms": 115.27896550244753,
"median_tpot_ms": 138.60413584286187,
"std_tpot_ms": 40.596832751555446,
"p99_tpot_ms": 162.54766270021474,
"mean_itl_ms": 115.27987397513024,
"median_itl_ms": 40.95466900616884,
"std_itl_ms": 319.0612322607716,
"p99_itl_ms": 1441.3871486997234
}
q3_235b_before.json
{
"date": "20260707-121446",
"endpoint_type": "openai",
"backend": "openai",
"label": null,
"model_id": "Qwen/Qwen3-235B-A22B-FP8",
"tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
"num_prompts": 1000,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": null,
"duration": 36.708394799847156,
"completed": 1000,
"failed": 0,
"total_input_tokens": 1024000,
"total_output_tokens": 128000,
"request_throughput": 27.241725100008015,
"request_goodput": null,
"output_throughput": 3486.940812801026,
"total_token_throughput": 31382.467315209236,
"max_output_tokens_per_s": 16747.0,
"max_concurrent_requests": 1000,
"rtfx": 0.0,
"mean_ttft_ms": 14843.877105322434,
"median_ttft_ms": 13492.701738374308,
"std_ttft_ms": 8986.719791165599,
"p99_ttft_ms": 31870.7056954992,
"mean_tpot_ms": 130.32083874423145,
"median_tpot_ms": 145.98531490943475,
"std_tpot_ms": 43.70416097908902,
"p99_tpot_ms": 182.12104581573274,
"mean_itl_ms": 130.32186564091228,
"median_itl_ms": 45.630548149347305,
"std_itl_ms": 339.3068440593827,
"p99_itl_ms": 1465.8603100758046
}
