The Transformers modeling backend is now as fast as native vLLM

Community Article
Published July 17, 2026

vLLM PR #47187 (merged, @ 5bce653)

Run any Hugging Face model in vLLM via --model-impl transformers and get the throughput of a hand-written native implementation — from a 4B dense model to a 235B-parameter FP8 MoE. Measured on 8×H100.

Transformers modeling backend vs native vLLM

Results (% of native vLLM)

Model Config Before PR After PR vs native
Qwen3-4B Dense · bf16 · TP1 98.7% 100.0% parity
Qwen3-32B Dense · bf16 · TP2 98.4% 100.1% parity
Qwen3-235B-A22B-FP8 MoE · FP8 · DP8+EP 96.6% 102.0% +2.0%

Each model is compared under three code paths on identical hardware/workload: native (--model-impl vllm, the bar to match), after (--model-impl transformers with the PR), and before (the same backend on the PR's parent commit).

Why the 235B MoE gains most

The PR routes Hugging Face MoE blocks through vLLM's FusedMoE — a grouped all-to-all expert dispatch the pre-PR naive per-expert loop can't match. At DP8 + expert parallel (the throughput-optimal MoE config) that's +5.6% over the pre-PR path, and still native parity. The QKV fusion also fixes a TP8+EP load failure — pre-PR the unfused k_proj shards to 64, not a multiple of the FP8 quant block (128).

Reproduce

Workload: random 1024-in / 128-out, 1000 prompts, torch.compile on, 8×H100. Point VLLM/VENV at your vLLM checkout (on PR #47187) and its venv, then ./benchmark.sh (or ./benchmark.sh q3_235b for one model).

Runner

benchmark.sh
#!/usr/bin/env bash
# =============================================================================
#  Reproduce the benchmark behind vLLM PR #47187
#  "Make the Transformers modeling backend as fast as native vLLM"
# =============================================================================
#
#  What this measures
#  ------------------
#  For each model we compare THREE conditions on identical hardware, workload
#  and flags — the only thing that changes is the code path:
#
#    native  --model-impl vllm          vLLM's hand-written model  (the bar to match)
#    after   --model-impl transformers  generic Transformers backend, WITH  PR #47187
#    before  --model-impl transformers  generic Transformers backend, WITHOUT PR #47187
#
#  PR #47187 is 100% Python and lives entirely under
#  vllm/model_executor/models/transformers/ — no CUDA/C++/build changes. So the
#  honest before/after delta is obtained by `git checkout`-ing just those files
#  to the PR's parent commit and back. No recompile, nothing else moves.
#
#  Every number the graphic reports is a WITHIN-model, WITHIN-harness ratio
#  (% of native, and after/before), so the comparison is apples-to-apples even
#  though absolute tok/s differ across model sizes and harnesses.
#
#  Hardware used for the published figures: 8x NVIDIA H100 80GB.
#
#  Usage
#  -----
#    ./benchmark.sh                # run every condition for every model
#    ./benchmark.sh q3_235b        # run only conditions whose tag matches this
#  Results (JSON + logs) land in ./results/. Re-running skips completed configs.
# =============================================================================
set -uo pipefail

# --- configure these two paths for your machine ------------------------------
VLLM=${VLLM:-$HOME/vllm}                 # editable vLLM checkout on PR #47187
VENV=${VENV:-$HOME/.venv/bin}            # venv with that vLLM installed
# -----------------------------------------------------------------------------
PR=5bce653e09                                 # PR #47187 merge commit
PARENT="${PR}^"                               # its parent = "before"
TFDIR=vllm/model_executor/models/transformers/
OUT="$(cd "$(dirname "$0")" && pwd)/results"
PORT=8231
FILTER="${1:-}"                               # optional tag substring filter
mkdir -p "$OUT"

# Shared workload for every run: 1024-token prompts, 128-token generations.
INLEN=1024; OUTLEN=128; NPROMPTS=1000

# Model matrix that produced the graphic.
#   tag | model | harness | tp | gpus | extra-flags
# harness = "throughput" (offline) or "serve" (online; required for data parallel).
MODELS=(
  "q3_4b|Qwen/Qwen3-4B|throughput|1|0|"
  "q3_32b|Qwen/Qwen3-32B|throughput|2|0,1|"
  "q3_235b|Qwen/Qwen3-235B-A22B-FP8|serve|8|0,1,2,3,4,5,6,7|--data-parallel-size 8 --enable-expert-parallel --max-model-len 8192"
)

# Always leave the tree back on the PR files, whatever happens.
restore(){ git -C "$VLLM" checkout "$PR" -- "$TFDIR" 2>/dev/null; }
trap restore EXIT

# Node-local compile caches on tmpfs. On a networked filesystem the multi-rank
# inductor/triton autotune cache races and throws "CUDA driver error: file not
# found"; keeping caches on /dev/shm with single-threaded compile avoids it.
setcache(){
  local tag=$1 c=/dev/shm/benchcache/$1
  rm -rf "$c"; mkdir -p "$c"/{triton,inductor,vllm}
  export TRITON_CACHE_DIR="$c/triton" TORCHINDUCTOR_CACHE_DIR="$c/inductor"
  export VLLM_CACHE_ROOT="$c/vllm"    TORCHINDUCTOR_COMPILE_THREADS=1
}

free_gpus(){  # wait until the GPUs are actually released before the next run
  pkill -9 -f "EngineCore" 2>/dev/null; pkill -9 -f "vllm serve" 2>/dev/null
  pkill -9 -f "VLLM::" 2>/dev/null
  for _ in $(seq 1 90); do
    local u; u=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits \
                 | awk '{s+=$1} END{print s+0}')
    [ "$u" -lt 3000 ] && break; sleep 2
  done
}

# ---- offline throughput harness (dense models) ------------------------------
run_throughput(){ # tag model impl tp gpus extra...
  local tag=$1 model=$2 impl=$3 tp=$4 gpus=$5; shift 5
  local json="$OUT/$tag.json" log="$OUT/$tag.log"
  grep -q tokens_per_second "$json" 2>/dev/null && { echo "SKIP $tag (done)"; return; }
  setcache "$tag"
  echo ">>> $tag : throughput impl=$impl tp=$tp gpus=$gpus"
  CUDA_VISIBLE_DEVICES=$gpus VLLM_LOGGING_LEVEL=INFO timeout 5400 \
    "$VENV/vllm" bench throughput \
      --model "$model" --model-impl "$impl" --tensor-parallel-size "$tp" \
      --dataset-name random --input-len $INLEN --output-len $OUTLEN \
      --num-prompts $NPROMPTS --output-json "$json" "$@" > "$log" 2>&1
  echo -n "    "; grep -o '"tokens_per_second":[^,]*' "$json" 2>/dev/null || echo "NO RESULT"
  echo    "    fused-op log lines: $(grep -cE '^Fused:' "$log")"
  free_gpus
}

# ---- online serving harness (data-parallel MoE) -----------------------------
run_serve(){ # tag model impl tp gpus extra...
  local tag=$1 model=$2 impl=$3 tp=$4 gpus=$5; shift 5
  local json="$OUT/$tag.json" slog="$OUT/${tag}_server.log" clog="$OUT/${tag}_client.log"
  grep -q total_token_throughput "$json" 2>/dev/null && { echo "SKIP $tag (done)"; return; }
  setcache "$tag"; rm -f "$json"
  echo ">>> $tag : serve impl=$impl gpus=$gpus"
  CUDA_VISIBLE_DEVICES=$gpus VLLM_LOGGING_LEVEL=INFO \
    "$VENV/vllm" serve "$model" --model-impl "$impl" --port $PORT "$@" > "$slog" 2>&1 &
  local spid=$! ready=0
  for _ in $(seq 1 400); do
    [ "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$PORT/health)" = 200 ] \
      && { ready=1; break; }
    kill -0 $spid 2>/dev/null || { echo "    server died (see $slog)"; break; }
    sleep 3
  done
  if [ "$ready" = 1 ]; then
    echo "    server ready -> client"
    "$VENV/vllm" bench serve --model "$model" --host 127.0.0.1 --port $PORT \
      --dataset-name random --random-input-len $INLEN --random-output-len $OUTLEN \
      --num-prompts $NPROMPTS --ignore-eos --save-result --result-filename "$json" > "$clog" 2>&1
    echo -n "    "; grep -oE '"total_token_throughput":[^,}]*' "$json" 2>/dev/null || echo "NO RESULT"
    echo    "    fused-op log lines: $(grep -cE '^Fused:' "$slog")"
  fi
  kill -9 $spid 2>/dev/null; free_gpus
}

dispatch(){ # spec impl condtag
  IFS='|' read -r tag model harness tp gpus extra <<< "$1"
  [ -n "$FILTER" ] && [[ "$tag" != *"$FILTER"* ]] && return
  # shellcheck disable=SC2086
  run_"$harness" "${tag}_$3" "$model" "$2" "$tp" "$gpus" $extra
}

# === Phase A: native + after — run on the PR files ===========================
echo "######## PHASE A: native + after (PR #47187 files) ########"
git -C "$VLLM" checkout "$PR" -- "$TFDIR"
for spec in "${MODELS[@]}"; do
  dispatch "$spec" vllm         native
  dispatch "$spec" transformers after
done

# === Phase B: before — swap ONLY the transformers/ dir to the parent commit ==
echo "######## PHASE B: before (parent commit ${PARENT}) ########"
git -C "$VLLM" checkout "$PARENT" -- "$TFDIR"
git -C "$VLLM" status --porcelain "$TFDIR"
for spec in "${MODELS[@]}"; do
  dispatch "$spec" transformers before
done

restore
echo "######## DONE — results in $OUT ; tree restored to PR files ########"

Raw results (vllm bench output)

q3_4b_native.json
{
  "elapsed_time": 24.27075677085668,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 41.201846709648486,
  "tokens_per_second": 47464.52740951506
}
q3_4b_after.json
{
  "elapsed_time": 24.28168404311873,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 41.18330500570834,
  "tokens_per_second": 47443.16736657601
}
q3_4b_before.json
{
  "elapsed_time": 24.58867740095593,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 40.6691252113106,
  "tokens_per_second": 46850.83224342981
}
q3_32b_native.json
{
  "elapsed_time": 79.22332654520869,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 12.622544944882506,
  "tokens_per_second": 14541.171776504647
}
q3_32b_after.json
{
  "elapsed_time": 79.12049886095338,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 12.63894963247645,
  "tokens_per_second": 14560.06997661287
}
q3_32b_before.json
{
  "elapsed_time": 80.50363391195424,
  "num_requests": 1000,
  "total_num_tokens": 1152000,
  "requests_per_second": 12.421799506512796,
  "tokens_per_second": 14309.913031502741
}
q3_235b_native.json
{
  "date": "20260707-115114",
  "endpoint_type": "openai",
  "backend": "openai",
  "label": null,
  "model_id": "Qwen/Qwen3-235B-A22B-FP8",
  "tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
  "num_prompts": 1000,
  "request_rate": "inf",
  "burstiness": 1.0,
  "max_concurrency": null,
  "duration": 35.44470150419511,
  "completed": 1000,
  "failed": 0,
  "total_input_tokens": 1024000,
  "total_output_tokens": 128000,
  "request_throughput": 28.212961530558907,
  "request_goodput": null,
  "output_throughput": 3611.25907591154,
  "total_token_throughput": 32501.331683203858,
  "max_output_tokens_per_s": 16680.0,
  "max_concurrent_requests": 1000,
  "rtfx": 0.0,
  "mean_ttft_ms": 15171.714233175619,
  "median_ttft_ms": 13715.512390015647,
  "std_ttft_ms": 9169.5805137529,
  "p99_ttft_ms": 30926.241073654965,
  "mean_tpot_ms": 116.817842786138,
  "median_tpot_ms": 138.0034470041539,
  "std_tpot_ms": 40.08971847574921,
  "p99_tpot_ms": 160.9097329545656,
  "mean_itl_ms": 116.81968330958155,
  "median_itl_ms": 41.13172448705882,
  "std_itl_ms": 319.89249689586126,
  "p99_itl_ms": 1439.9876020080408
}
q3_235b_after.json
{
  "date": "20260707-120324",
  "endpoint_type": "openai",
  "backend": "openai",
  "label": null,
  "model_id": "Qwen/Qwen3-235B-A22B-FP8",
  "tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
  "num_prompts": 1000,
  "request_rate": "inf",
  "burstiness": 1.0,
  "max_concurrency": null,
  "duration": 34.749073383864015,
  "completed": 1000,
  "failed": 0,
  "total_input_tokens": 1024000,
  "total_output_tokens": 128000,
  "request_throughput": 28.77774578197407,
  "request_goodput": null,
  "output_throughput": 3683.551460092681,
  "total_token_throughput": 33151.96314083412,
  "max_output_tokens_per_s": 16526.0,
  "max_concurrent_requests": 1000,
  "rtfx": 0.0,
  "mean_ttft_ms": 14851.332848654361,
  "median_ttft_ms": 12809.239127556793,
  "std_ttft_ms": 9119.170769998696,
  "p99_ttft_ms": 30412.630736478604,
  "mean_tpot_ms": 115.27896550244753,
  "median_tpot_ms": 138.60413584286187,
  "std_tpot_ms": 40.596832751555446,
  "p99_tpot_ms": 162.54766270021474,
  "mean_itl_ms": 115.27987397513024,
  "median_itl_ms": 40.95466900616884,
  "std_itl_ms": 319.0612322607716,
  "p99_itl_ms": 1441.3871486997234
}
q3_235b_before.json
{
  "date": "20260707-121446",
  "endpoint_type": "openai",
  "backend": "openai",
  "label": null,
  "model_id": "Qwen/Qwen3-235B-A22B-FP8",
  "tokenizer_id": "Qwen/Qwen3-235B-A22B-FP8",
  "num_prompts": 1000,
  "request_rate": "inf",
  "burstiness": 1.0,
  "max_concurrency": null,
  "duration": 36.708394799847156,
  "completed": 1000,
  "failed": 0,
  "total_input_tokens": 1024000,
  "total_output_tokens": 128000,
  "request_throughput": 27.241725100008015,
  "request_goodput": null,
  "output_throughput": 3486.940812801026,
  "total_token_throughput": 31382.467315209236,
  "max_output_tokens_per_s": 16747.0,
  "max_concurrent_requests": 1000,
  "rtfx": 0.0,
  "mean_ttft_ms": 14843.877105322434,
  "median_ttft_ms": 13492.701738374308,
  "std_ttft_ms": 8986.719791165599,
  "p99_ttft_ms": 31870.7056954992,
  "mean_tpot_ms": 130.32083874423145,
  "median_tpot_ms": 145.98531490943475,
  "std_tpot_ms": 43.70416097908902,
  "p99_tpot_ms": 182.12104581573274,
  "mean_itl_ms": 130.32186564091228,
  "median_itl_ms": 45.630548149347305,
  "std_itl_ms": 339.3068440593827,
  "p99_itl_ms": 1465.8603100758046
}

Community

Sign up or log in to comment