ndaly/Mistral-Small-4-119B-2603

mistral-small-4-119b-p300x2 on Tenstorrent p300x2

Derived from mistralai/Mistral-Small-4-119B-2603 by Mistral AI, unmodified. See the original model card for training and evaluation details.

Mistral Small 4 (Mistral-Small-4-119B-2603), a 119B-parameter mixture-of-experts transformer with 128 experts, 4 active and 6.5B active parameters per token, and multi-head latent attention, served with vLLM in instruct mode. Text only, tested at one user and at 32 concurrent users.

Runs on Tenstorrent p300x2 (mesh P300x2) — 1,048,576-token context, up to 32 concurrent sequences.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p300x2
Context	1,048,576 tokens
License	apache-2.0

Intended use

General chat assistant and coding use through the OpenAI-compatible chat and completions API, in the languages the checkpoint covers (English, French, German, Spanish, Portuguese, Italian, Chinese, Japanese, Korean, Arabic and others).

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300x2
• 121 GB of free space in the Hugging Face cache for the weight shards, and about 100 GB more under ~/.cache/tt-model/ for the converted weights
• Hugepages configured on the host

Quickstart

tt serve ndaly/Mistral-Small-4-119B-2603

tt serve downloads the Docker image and the mistralai/Mistral-Small-4-119B-2603 at a11f36bebf709121056b1dbcc943d1c6afbe494d weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve ndaly/Mistral-Small-4-119B-2603.

The first boot also converts the weights for the four-chip mesh, which takes tens of minutes; later boots load the converted weights from the cache.

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Mistral-Small-4-119B-2603" — the weights id, not this package's name.

The chat template adds a default system prompt of about 564 tokens to every chat completion; it is included in the time to first token below.

Expected performance

eval	this package	reference	prompts
IFEval, prompt-strict (1,024-token cap)	72.83 %	—	541
HumanEval-instruct, pass@1	92.68 %	—	164

Evals ran through the served endpoint. The original model card reports its benchmarks as charts, with no IFEval or HumanEval figure.

profile	ISL	OSL	concurrency	N	TTFT (ms)	prefill tok/s/u	decode tok/s/u	E2EL (s)
default	128 (691 served with the default system prompt)	128	1	8	1,544	448	45.9	4.3
default	100 (about 660 served with the default system prompt)	100	32	32	39,609	17	2.1 (66 aggregate)	48.4

Means over the N requests of each row: vllm bench serve with random-token prompts of the stated length, rendered through the chat template (which adds the default system prompt), the stated output length requested but not pinned (a reply may stop earlier), measured on one p300x2 through the served chat completions endpoint on 2026-09-16. Prefill tok/s/u is served prompt tokens / TTFT (at 32 users the 32 prefills run one after another, so TTFT covers the wait); decode tok/s/u is 1 / TPOT at one user and the aggregate output rate divided by the user count at 32 users. Decode is 21 to 22 ms per token for one user; prefill runs at about 2 ms per prompt token up to the full 1,048,576-token context.

Limitations

• Validated on p300x2 only (one four-chip mesh, tensor-parallel over 4 chips); up to 32 concurrent sequences.
• Not for image input: the checkpoint's vision input is not ported (text only). Not for tool calling or reasoning mode: no tool-call or reasoning parser is configured, so tools in a request are not turned into tool_calls and the checkpoint's reasoning mode is not validated.
• Prefix caching and chunked prefill are off, so every request prefills from scratch and prefill is sequential across users; at 32 users aggregate decode is about 66 tok/s.
• Prompts up to 1,048,576 tokens are accepted, at about 2 ms per prompt token (around 35 minutes for a full-context prompt). Accuracy beyond the checkpoint's 262,144-token training context was not evaluated.
• GPQA was not measured.
• Weights run at a reduced-precision block-float policy, not bf16; outputs differ from the Hugging Face reference.

Licensing

Weights: Apache-2.0 (Mistral AI). Port and serving code in code/: Apache-2.0.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/ndaly/Mistral-Small-4-119B-2603/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
vLLM	vllm-0.1.dev14195+g8c28fcecb.d20260916.empty-py3-none-any.whl — a wheel the author built
vllm-tt-plugin	a local checkout — commit not published (dirty tree — the image includes uncommitted changes)
code/ digest	a4dc23aeb6810dad (sha256, first 16 hex digits)
built	2026-09-16T16:55:50+00:00 by tt-model 0.1.0