devstral-small-2-24b-instruct-2512 on Tenstorrent p300 and p150x4 and p300x2

Derived from mistralai/Devstral-Small-2-24B-Instruct-2512, weights revision 55c5b41e. This is that model's implementation on Tenstorrent hardware; see the original model card for training and evaluation details.

Devstral Small 2 is Mistral's 24B agentic coding model, tuned for software engineering work such as exploring a repository, editing several files and driving tools. It is a 24B dense decoder-only transformer whose FP8 E4M3 weights are dequantised to bfloat16 at load. This package serves it on Tenstorrent Blackhole through vLLM with tool calling enabled, as an experimental community bring-up. It is text-only: the checkpoint also carries a Pixtral vision tower, which this package does not serve. Three profiles — p300, p150x4 and p300x2 — all serve the text decoder, and all three were benchmarked. Two devices minimum. Measured on 2 x Blackhole p300c.

Runs on Tenstorrent p300 or p150x4 or p300x2 — see the serving profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p150x4, p300, p300x2
Context	65,536 tokens
License	Apache 2.0

Intended use

Agentic coding over an OpenAI-compatible API — repository exploration, multi-file edits, and tool-driven workflows — for one user or a small batch on a single four-device Blackhole host.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p300 or p150x4 or p300x2
• The TT CLI (tt, which wraps tt-model 5.1) and a working Docker installation on a Tenstorrent Blackhole host with at least two devices. The image is pulled and run by tt model pull and tt serve; the weights are downloaded separately on first serve.

Quickstart

tt serve anirud/devstral-small-2-24b-instruct-2512

tt serve downloads the Docker image and the mistralai/Devstral-Small-2-24B-Instruct-2512 at 55c5b41e98c2dbd21b0c8afffc540dcfc9eb5128 weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve anirud/devstral-small-2-24b-instruct-2512.

Validated on 2 x Blackhole p300c (4 devices), with 2, 4 and 4 devices opened for the p300, p150x4 and p300x2 profiles. First boot compiles kernels and takes about 360 s before the server accepts requests.

Loading this checkpoint's tokenizer through the Hugging Face backend prints a transformers warning about an "incorrect regex pattern". It is spurious for this checkpoint and needs no action: the pattern shipped in tokenizer.json is already the corrected one, loading with fix_mistral_regex=True produces identical ids over 5,002 probes, and the two tokenizer backends agree on 5,386 further probes with no differences. The served package resolves the Mistral backend and does not print it.

Serving profiles

Select a different serving profile using --profile.

profile	hardware	mesh	max_num_seqs	max_model_len	description
p300	p300	P300	4	32768	Two dies, one p300 card.
p150x4 (default)	p150x4	P150x4	8	65536	Four dies as a line. The default.
p300x2	p300x2	P300x2	32	65536	The same four dies, labelled as two p300 cards (a QB2).

Capabilities

The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "mistralai/Devstral-Small-2-24B-Instruct-2512" — the weights id, not this package's name.

Tool calling is enabled (--tool-call-parser mistral): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.

A complete request, with the sampling recipe this checkpoint expects:

curl http://127.0.0.1:20000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mistralai/Devstral-Small-2-24B-Instruct-2512",
    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],
    "temperature": 0.15
  }'

Image input is not supported. The checkpoint carries a Pixtral vision tower, but this package serves the language model only; a request carrying image data is rejected rather than silently answered as if the image were not there.

Tool calling is enabled with the Mistral parser. Send an OpenAI tools payload and the reply carries finish_reason: tool_calls with the arguments as JSON; return the result as a role: tool message to continue the exchange. This is not a reasoning model, so no reasoning parser is configured and no thinking block is emitted.

Expected performance

Closed-loop latency sweep of the three text profiles, measured 2026-10-07 on 2 x Blackhole p300c (4 devices) with 2, 4 and 4 devices opened for p300, p150x4 and p300x2, served from this image with tt-model serve and driven by vllm bench serve 0.26.0 inside the container, at concurrencies of 1 to 32 streams. Prompts are random token ids of exactly ISL tokens; output is pinned to OSL tokens with ignore_eos, so every row holds the batch full for its whole length — these are sustained-rate numbers, not natural-conversation ones. One unrecorded warm-up request precedes each input length. N = 4 x users up to 4k input, 2 x users up to 16k, and users beyond. TTFT is the time to the first streamed token; TPOT = (E2EL - TTFT) / (OSL - 1). Values are medians over the row. A 300 s median-E2EL fence was configured to stop the ladder; no row reached it, the slowest measured median E2EL being 277.3 s at 32,768 input and 32 users on p300x2. Characterization only, no configured targets. First boot compiles kernels and takes about 360 s; that is startup, not inference, and is excluded from every row.

Column key: ISL/OSL = input/output tokens; Users = concurrent streams; N = prompts in the row; TTFT = time to first token; Prefill = ISL / TTFT, derived from the two columns either side of it rather than measured directly, so at higher concurrency it carries the queueing delay inside TTFT and understates the device's prefill rate; TPOT = time per output token; Decode = output tokens per second per user; Output = output tokens per second across all users; E2EL = end-to-end latency.

One row per serving profile, single stream at 128/128. The full ladder for each profile follows.

Profile	ISL	OSL	Concurrency	N	TTFT (ms)	Prefill (tok/s/u)	Decode (tok/s/u)	E2EL (s)
p300	128	128	1	4	215	595	21.2	6.21
p150x4	128	128	1	4	132	970	28.2	4.62
p300x2	128	128	1	4	131	977	26.5	4.90

p150x4 — (1, 4) mesh, 65,536-token context; 20 of 20 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	Prefill (tok/s/u)	TPOT (ms)	Decode (tok/s/u)	Out (tok/s)	E2EL (s)	Status
128	128	1	4	132	970	35.5	28.2	28	4.62	ok
128	128	2	8	178	719	37.0	27.0	52	4.89	ok
128	128	4	16	446	287	42.3	23.7	88	5.78	ok
128	128	8	32	729	176	49.5	20.2	146	7.08	ok
1,024	256	1	4	139	7,367	36.0	27.8	28	9.32	ok
1,024	256	2	8	194	5,278	38.1	26.2	52	9.86	ok
1,024	256	4	16	368	2,783	42.5	23.5	92	11.13	ok
1,024	256	8	32	533	1,921	49.8	20.1	151	13.54	ok
4,096	256	1	4	497	8,241	37.1	26.9	26	9.86	ok
4,096	256	2	8	631	6,491	39.4	25.4	48	10.68	ok
4,096	256	4	16	704	5,818	43.8	22.8	81	12.39	ok
4,096	256	8	32	859	4,768	52.1	19.2	124	15.85	ok
16,384	256	1	2	1,444	11,346	41.9	23.9	21	12.12	ok
16,384	256	2	4	1,855	8,832	44.1	22.7	34	14.87	ok
16,384	256	4	8	2,009	8,155	48.1	20.8	51	20.22	ok
16,384	256	8	16	10,478	1,564	57.2	17.5	48	33.55	ok
32,768	256	1	1	318	103,044	47.2	21.2	21	12.35	ok
32,768	256	2	2	3,935	8,327	63.4	15.8	25	20.10	ok
32,768	256	4	4	8,303	3,947	93.5	10.7	25	32.15	ok
32,768	256	8	8	6,452	5,079	259.8	3.8	22	72.71	ok

p300x2 — (1, 4) mesh, 65,536-token context; 30 of 30 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	Prefill (tok/s/u)	TPOT (ms)	Decode (tok/s/u)	Out (tok/s)	E2EL (s)	Status
128	128	1	4	131	977	37.7	26.5	26	4.90	ok
128	128	2	8	178	719	39.7	25.2	49	5.23	ok
128	128	4	16	446	287	44.6	22.4	83	6.13	ok
128	128	8	32	730	175	51.6	19.4	140	7.29	ok
128	128	16	64	1,363	94	73.5	13.6	191	10.70	ok
128	128	32	128	1,771	72	117.5	8.5	234	17.50	ok
1,024	256	1	4	138	7,420	39.1	25.5	25	10.12	ok
1,024	256	2	8	191	5,361	41.2	24.2	48	10.70	ok
1,024	256	4	16	373	2,745	45.5	22.0	85	11.98	ok
1,024	256	8	32	486	2,107	53.5	18.7	142	14.49	ok
1,024	256	16	64	914	1,120	74.3	13.5	198	20.69	ok
1,024	256	32	128	1,791	572	118.9	8.4	239	34.19	ok
4,096	256	1	4	526	7,787	43.6	22.9	22	11.62	ok
4,096	256	2	8	698	5,868	45.7	21.9	41	12.36	ok
4,096	256	4	16	778	5,265	50.2	19.9	72	14.34	ok
4,096	256	8	32	933	4,390	58.0	17.3	114	18.01	ok
4,096	256	16	64	5,665	723	92.3	10.8	138	30.30	ok
4,096	256	32	128	12,445	329	164.7	6.1	138	54.23	ok
16,384	256	1	2	1,443	11,354	61.5	16.2	15	17.14	ok
16,384	256	2	4	1,853	8,842	63.4	15.8	26	19.79	ok
16,384	256	4	8	2,005	8,172	67.5	14.8	41	25.10	ok
16,384	256	8	16	2,045	8,012	80.5	12.4	55	35.64	ok
16,384	256	16	32	53,193	308	146.9	6.8	35	96.81	ok
16,384	256	32	64	126,224	130	136.4	7.3	36	157.21	ok
32,768	256	1	1	317	103,369	85.0	11.8	12	21.99	ok
32,768	256	2	2	2,826	11,595	95.1	10.5	19	27.08	ok
32,768	256	4	4	17,966	1,824	92.1	10.9	25	41.46	ok
32,768	256	8	8	16,683	1,964	91.6	10.9	24	55.20	ok
32,768	256	16	16	131,445	249	150.1	6.7	16	169.96	ok
32,768	256	32	32	238,627	137	150.0	6.7	17	277.29	ok

p300 — (1, 2) mesh, 32,768-token context; 12 of 12 planned rows measured

ISL	OSL	Users	N	TTFT (ms)	Prefill (tok/s/u)	TPOT (ms)	Decode (tok/s/u)	Out (tok/s)	E2EL (s)	Status
128	128	1	4	215	595	47.2	21.2	21	6.21	ok
128	128	2	8	281	456	48.5	20.6	40	6.46	ok
128	128	4	16	727	176	52.1	19.2	69	7.38	ok
1,024	256	1	4	224	4,571	47.3	21.2	21	12.28	ok
1,024	256	2	8	308	3,325	48.9	20.4	40	12.73	ok
1,024	256	4	16	605	1,693	52.9	18.9	72	14.08	ok
4,096	256	1	4	802	5,107	48.5	20.6	20	12.95	ok
4,096	256	2	8	1,029	3,981	50.2	19.9	36	13.99	ok
4,096	256	4	16	1,129	3,628	55.2	18.1	63	16.12	ok
16,384	256	1	2	1,691	9,689	52.0	19.2	17	14.96	ok
16,384	256	2	4	2,977	5,504	53.9	18.5	24	21.04	ok
16,384	256	4	8	3,798	4,314	60.8	16.5	32	32.12	ok

Evaluations

Run with lm_eval 0.4.10.dev0 (the Tenstorrent tstescoTT/lm-evaluation-harness evals-common fork) against this package's own serving stack on the p150x4 profile, sampling at the checkpoint's own recipe (temperature 0.15). No reference run was available, so these are absolute Tenstorrent figures with no delta: the upstream card publishes only agentic results (SWE-Bench human-validated split 68.0%, SWE-Bench Multilingual 55.7%, Terminal Bench 2 22.5%), and lm_eval cannot run those harnesses.

• HumanEval, instruct format — 88.4% (145 of 164), all 164 problems answered; temperature 0.15, scored as pass@1,create_test.
• HumanEval, raw completion — 39.0% (64 of 164), all 164 answered; same recipe through the completions endpoint.
• LiveCodeBench v6 — 32.0%, 174 of 175 answered; temperature 0.15, 2,048 new tokens, scored as acc,none.

The two HumanEval rows are the same benchmark on the same build, 49 points apart, and the difference is protocol alone. Posed as a chat turn the model scores 88.4%; asked to continue a bare function signature it scores 39.0% and sometimes answers in the wrong register. The instruct row reflects how an instruction-tuned model is normally used; the raw-completion row is the protocol HumanEval defines. Both are reported.

Against a CPU reference of this same checkpoint, short closed-form answers match exactly while free-form code diverges token-for-token and stays functionally correct: a bfloat8_b device does not reproduce an fp32 host run verbatim in open-ended generation.

Sampling

The checkpoint's generation_config.json sets temperature 0.15 and the upstream card uses that value in every example; it is the recipe above. This package serves with sample_on_device_mode: all, so sampling runs on the device rather than the host.

Limitations

Image input is not supported, and no multimodal endpoint is exposed. The checkpoint carries a Pixtral vision tower; serving it was prototyped on this hardware and held back, because on the prototype profile a text request that followed a long generation could return a wrong answer with nothing logged. That profile is not in this package, and the three profiles here were never affected (0 of 48 wrong under the test that reproduced it). Requests carrying image data are refused rather than quietly answered as text.

There is no single-device profile, and two devices is the minimum. The weights and KV cache fit in one 32 GB device, but a prefill matmul overruns L1 by about 56 KB: tt_transformers picks the same MLP prefill grids on one device as on four, so one device's cores carry the whole MLP instead of a quarter of it. The configuration does not boot and is not declared.

max_model_len is a configured serving limit, not the model's. The upstream checkpoint advertises a 256k context; this package configures 32,768 tokens on p300 and 65,536 on p150x4 and p300x2. Longer contexts were not validated.

The performance tables and the evaluations were measured on the 2026-10-07 build of this package; every row completed, and the evaluations were run on p150x4 only. This image is a later build of the same three profiles. The single-stream 128/128 row on p150x4 was re-measured on it: TTFT 136.4 ms against 132, decode 27.7 tok/s/u against 28.2, E2EL 4.71 s against 4.62 — within run-to-run noise on N = 4. The full ladder was not re-run.

Two numerical paths differ from a reference implementation and could change results without raising an error: the FP8 E4M3 checkpoint is dequantised to bfloat16 at load using its per-tensor scales, and the Llama-4 style query scaling is computed on the device. Both were checked against a host reference — the query scale matched at every position tested, the dequantisation was asserted against an independent implementation — but accumulated bfloat16 drift over long generations is not bounded by those checks. Free-form generation is correspondingly not reproducible token-for-token against a host run of the same checkpoint; compare behaviour, not tokens.

The image cannot be rebuilt from public source. The tt-metal branch it was built from lives on the build host and has not been pushed, so provenance reports the commit as not published rather than linking it. The contents are shipped in code/ and are byte-identical to the model code inside the image.

tt-model info cannot resolve a local bundle directory — it accepts only a Hugging Face repo id — so inspecting a staged package before publishing is not possible with it.

Licensing

The weights are Apache-2.0, as declared by the upstream repository; the upstream repository is not gated and ships no separate LICENSE file. The tt-metal model code in code/ is Apache-2.0. No non-commercial or redistribution restriction applies to either.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/anirud/devstral-small-2-24b-instruct-2512/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli).

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	a local checkout — commit not published
vLLM	v0.26.0
vllm-tt-plugin	a local checkout — commit not published
code/ digest	e5670ebe2bc89243 (sha256, first 16 hex digits)
built	2026-10-19T13:20:47+00:00 by tt-model 0.1.0