qwen3.8-flash-next-p150x8

Qwen/Qwen3.8-Flash-Next (51.2B MoE, hybrid gated delta-net + sparse attention) on a Tenstorrent P150x8 (8× Blackhole P150), served by the port's own OpenAI-compatible server.

• TP8 for the dense path, EP8 for the 512 routed experts (BF4), 32K context.
• 32 decode slots: 28 concurrent users + 4 batched prefill lanes; a lone user takes a batch-1 fast path.
• Greedy output is bitwise repeatable; every start replays a CPU greedy record and must match 96/96.
• Sampling (temperature, top-p/k, min-p, penalties, seed), streaming, tool calls, reasoning_content, logprobs.
• Changed since the fifth publication (2026-09-19): the kernel round — MoE prefill in 512-token blocks (compact per-token routing metadata), a fused GDN recurrent-step kernel, a 960 MiB trace region holding the full 256/512/1024 slab ladder and a 2-step pass-fill wait (slab prefill 0.59 → 0.42 ms/token, decode step 72.3 → 69.7 ms) — 1,024/128 TTFT at one user 846 → 764 ms, 1,024/128 at 28 users 96.8 → 105.2 tok/s; the expert cache of the earlier images is reused (same cache key).

Quickstart

tt-model pull  changh95/qwen3.8-flash-next-p150x8 --with-weights   # image + 360 GB weights
tt-model serve changh95/qwen3.8-flash-next-p150x8

• First start: 35 min (converts the experts into a 107 GB cache under `/.cache/tt-model/`). Warm start: ~4-6 min.
• Thinking is on by default; pass enable_thinking: false for a direct answer.

curl -s http://localhost:20000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3.8-Flash-Next", "messages": [{"role": "user", "content": "Explain KV caching in two sentences."}], "max_tokens": 256, "enable_thinking": false}'

Benchmarks

Served from this container, 2026-09-20. Client: tt-inference-server's --workflow benchmarks recipe (vllm bench serve, random prompts of exactly ISL tokens, temperature 0, streaming). t/s = aggregate output tokens/s; t/s/u = OSL / mean end-to-end latency per user.

Throughput by concurrent users (t/s, t/s/u in parentheses)

ISL / OSL	1	2	4	8	16	28
128 / 128	20 (20.4)	26 (13.0)	51 (12.8)	92 (11.5)	149 (9.4)	205 (7.4)
1,024 / 128	19 (18.6)	25 (12.5)	42 (10.5)	64 (8.1)	88 (5.6)	105 (3.8)
2,048 / 128	17 (16.7)	23 (11.3)	35 (8.9)	50 (6.3)	64 (4.0)	72 (2.6)
4,096 / 128	15 (14.9)	19 (9.4)	27 (6.8)	35 (4.4)	41 (2.6)	44 (1.6)
128 / 1,024	22 (21.8)	30 (15.1)	59 (14.7)	111 (13.9)	203 (12.7)	322 (11.5)
1,024 / 1,024	21 (21.1)	29 (14.7)	55 (13.9)	101 (12.6)	176 (11.0)	260 (9.3)

tt-inference-server --workflow benchmarks grid

users	ISL	OSL	TTFT ms	TPOT ms	t/s	t/s/u
1	128	128	363	46.6	20.4	20.4
28	128	128	1,441	124.6	205.3	7.4
1	128	1,024	373	45.5	21.8	21.8
28	128	1,024	1,808	84.9	322.1	11.5
1	1,024	128	764	48.3	18.6	18.6
28	1,024	128	3,998	231.3	105.2	3.8
1	2,048	128	1,346	49.7	16.7	16.7
28	2,048	128	6,627	329.6	72.5	2.6
1	4,096	128	2,351	49.2	14.9	14.9
28	4,096	128	15,448	507.4	44.0	1.6
1	8,192	128	4,509	49.4	11.9	11.9
28	8,192	128	75,039	523.2	23.3	0.9
1	16,384	128	8,887	49.2	8.5	8.5
28	16,384	128	185,046	582.2	11.9	0.5

• Prefill is the bound on prompt-heavy traffic: ~0.5-0.7 ms per prompt token at one user (1K-16K prompts), and long prompts serialise at 28 users (the 8K/16K rows).
• 28 is the maximum concurrency: 4 of the 32 slots are prefill lanes, which beat the 32-slot no-lane server on every measured cell.

Provenance

• tt-metal fork sjettTT/tt-qwen-3.8-flash-next, branch qwen38-p150x8-batching, commit b8733f1f34fd1272610bbd55f6c750ec9bae7f38 (not yet pushed to GitHub; code/ in this repo is byte-identical to the model code in the image, digest fe58cf75b515d194).
• Weights: Qwen/Qwen3.8-Flash-Next @ f5d08274bafd880402bd16f5e3e6c514136ec06c.
• Built 2026-09-19 with tt-model-manager 0.1.0 (schema 5.1, kind tt-dit-server). Docs: code/models/demos/blackhole/qwen38_flash_next/docs/.