motif-3-bh-galaxy

Motif Technologies Motif-3 (314B MoE, 13.2B active, 384 experts + 1 shared, 1 MTP layer) served on one Tenstorrent Blackhole Galaxy (32 chips) through vLLM, with MTP self-speculative decoding.

Runs on p150x32 (mesh (4, 8), fabric FABRIC_2D_TORUS_XY): 32,768-token context, up to 32 concurrent sequences. Attention is 8-way tensor-parallel, the MoE 32-way expert-parallel.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5, self-contained bundle: no Docker needed).

Quickstart

export TT_MODEL_MODELS_DIR=/big/disk/tt-model   # ≈630 GB weights + ≈344 GB device weight cache
tt-model pull  changh95/motif-3-bh-galaxy --with-weights
tt-model serve changh95/motif-3-bh-galaxy

pull --with-weights installs the bundle and downloads the Motif-Technologies/Motif-3 weights at 2ed2ed5c (they are not in this repo). serve starts an OpenAI-compatible server on port 20000 (or the next free port); it is ready when it logs Application startup complete. The first start writes the device weight cache and compiles kernels (about 95 min); later starts take about 2 min.

Two launches, picked with MOTIF3_LAUNCH:

Launch	Command	Use it for
mtp (default)	tt-model serve changh95/motif-3-bh-galaxy	everything else: greedy and sampled requests both speculate (lossless)
plain	MOTIF3_LAUNCH=plain tt-model serve changh95/motif-3-bh-galaxy	top-N logprobs, logit_bias, min_p on sampled requests (refused on mtp), bursts of very short outputs at many users

Variants: MOTIF3_LAUNCH=plain MOTIF3_ASYNC_DECODE=off tt-model serve changh95/motif-3-bh-galaxy (synchronous decode), and tt-model serve changh95/motif-3-bh-galaxy --no-enable-prefix-caching / MOTIF3_LAUNCH=plain tt-model serve changh95/motif-3-bh-galaxy --no-enable-prefix-caching (prefix caching off, as benchmarked).

Requirements: SFPI 7.84.0 (sfpi_install.sh in the bundle installs it without sudo), and tt-smi off your PATH when running tt-model (tt-model-manager up to 7994e0a does not know the board type tt-galaxy-bh).

Performance

tt-inference-server run.py --workflow benchmarks on one Blackhole Galaxy with real weights (BFP8 experts and KV cache), warm caches, prefix caching off, on the tt-metal commit this bundle was built from (2026-10-11). Greedy: random prompts, ignore_eos, temperature 0. Rows are concurrent users, columns are input/output tokens per user. A dash: not measured. Raw reports: reports/benchmarks/.

mtp launch (default), greedy. MTP acceptance 0.86-0.92.

Per-user decode, tokens per second:

users \ in/out tokens	128/1024	1024/1024
1	53.7	51.5
2	50.9	48.0
4	46.1	42.9
8	40.0	36.2
16	35.0	27.2
32	26.6	21.2

Aggregate decode throughput, tokens per second:

users \ in/out tokens	128/1024	1024/1024
1	54	51
2	100	95
4	181	170
8	312	282
16	549	428
32	832	666

Time to first token, mean over users, ms:

users \ in/out tokens	128/1024	1024/1024
1	216	615
2	270	717
4	388	957
8	393	1,099
16	574	1,487
32	750	2,322

plain launch (asynchronous decode), greedy. TTFT ms / time per output token ms / aggregate tokens per second:

users \ in/out tokens	128/1024	1024/128
1	220 / 29.4 / 34	620 / 29.4 / 29
8	797 / 34.0 / 230	2,009 / 42.3 / 139
32	1,438 / 40.9 / 756	3,170 / 94.0 / 271

Sampled real text (TIS_BENCHMARK_VARIANT=sampled_text: SPEED-Bench prompts cut to 128 tokens, up to 1,024 output tokens, temperature 1.0, top-p 0.95). TTFT ms / time per output token ms / aggregate tokens per second:

users	mtp (speculative sampling, acceptance 0.69)	plain (synchronous)
1	235 / 20.7 / 48	237 / 31.5 / 32
8	432 / 27.6 / 245	561 / 37.5 / 186
32	680 / 43.4 / 658	792 / 48.6 / 574

Long prompts, 1 user (prefill is the same on both launches; measured 2026-10-08, prefill code unchanged since). Time to first token, s:

input tokens	1,024	2,048	4,096	8,192	16,384	32,640
TTFT	0.61	0.64	0.77	2.11	5.07	12.28

With prefix caching on (as shipped), 128/1024 time per output token: mtp 19.2 ms at 1 user and 35.6 ms at 32 users, plain 30.4 / 42.1 ms.

Evals

tt-inference-server run.py --workflow evals (lm-eval generation tasks, --limit-samples-mode ci-nightly subsets, plus repeats), sampled on the device at temperature 1.0, top-p 0.95, thinking on, outputs capped by the 32K context. Both launches at this bundle's code (2026-10-11). Raw reports: reports/evals/, analysis: reports/phaseG/evals/eval_analysis.json.

Task	Problems	mtp (speculative sampling)	plain (synchronous)
r1_math500	100	97.0	99.0
r1_aime24	75 (15 + 2 × 30)	82.7	88.0
r1_gpqa_diamond	120 (40 × 3)	85.0	86.7

• All differences between the launches are within sampling noise (|z| ≤ 1.01); both pass the GPQA Diamond bar of 79.23.
• Speculative sampling is lossless: a distribution test of mtp with and without it passes at 1 user and with batch-invariant prefill at 32 users, and seeded requests give the same tokens with and without it (reports/phaseG/distribution_test/).
• plain with asynchronous decode (as shipped) gives the same tokens as synchronous decode (reports/two_launch/).
• Greedy, for reference (mtp speculating; the same scores as without MTP): MATH-500 96.0, AIME24 80.0, GPQA Diamond 77.5. Greedy scores are lower than sampled on this model, so serve with sampling.

Provenance

The exact sources the bundle was built from. metal/models in this repo is byte-identical to the model code of the tt-metal commit below.

component	built from
tt-metal	571b979ae3f (branch motif3-opt-g, on upstream d611b1ec78d)
vLLM	v0.26.0
vllm-tt-plugin	9d746e6 (branch motif3-bh-galaxy, on upstream dd287f9)
tt-inference-server	5c25756e + the Motif-3 model specs in provenance/tt-inference-server-motif3.patch
weights	Motif-Technologies/Motif-3 @ 2ed2ed5c
metal/ digest	8c73676b5b8a5c3a (sha256, first 16 hex digits)
built	2026-10-12T08:35:49+00:00 by tt-model 0.1.0