episod/qwen3.8-27b-dflash2-p150

Qwen3.8-27B with DFlash2 on one Blackhole chip (v6 thin bundle, DRAFT)

Qwen/Qwen3.8-27B served with vLLM on 1 chip (p150-class, 32 GB), with DFlash2 speculative decoding (incoai/Qwen3.8-27B-DFlash2). Packaged as a v6 thin tt-model bundle, which is a beta, unsupported format: flags and layout may change. Not reviewed or endorsed by anyone but the person who built and measured it.

Hardware	1 chip (p150-class, 32 GB), TP=1; firmware bundle 19.15.0; tt-kmd 2.11.0
Context / users	16,384 tokens, up to 2 concurrent users (KV pool 16,512 tokens, bf8)
Decoding	DFlash2 speculative decoding, composite GDN verify (QWEN36_GDN_SPEC_FUSED=0), greedy only
Weights	Qwen/Qwen3.8-27B@1d4bf0f2 (bfp4 gate/up, bfp8 rest); drafter incoai/Qwen3.8-27B-DFlash2@dedf8df6

What this is good for, and what it is not

This is a chat-and-short-tasks model on a single card. It is not a coding-agent backend. A 27B model on one chip at roughly 3x plain single-chip decode speed is the point; the price is a 16,384-token context.

• Works: chat, Q&A, short code snippets and explanations, anything whose whole conversation stays well under 16K tokens.
• Does not fit: agentic coding tools. In a short hands-on trial by the person who built this, qwencode could not be used because its prompts do not fit in 16K tokens, and aider ran but was not practical for real coding work. Plain chat worked well. That was one person's informal trial, not a benchmark, and the tool versions and prompts were not recorded.
• If you need coding-agent context lengths, use a multi-chip build of the same model. The numbers below say nothing about agentic workloads.

Use

tt-model pull  episod/qwen3.8-27b-dflash2-p150
tt-model serve episod/qwen3.8-27b-dflash2-p150          # OpenAI-compatible server, model id Qwen/Qwen3.8-27B

The bundle builds its own venv (install.sh: interpreter, ttnn, tt-metal-models, vLLM 0.26.0 built for the empty target, the plugin). Host needs: a Blackhole chip with firmware and driver, 1G hugepages mounted at /dev/hugepages-1G, the SFPI toolchain (/opt/tenstorrent/sfpi), network access for the install, and roughly 60 GB of disk for weights plus tens of GB for caches. Tool calling (qwen3_coder) and reasoning (qwen3) parsers are enabled; temperature and sampling controls are not supported by this decoder.

What was verified on this bundle (2026-09-30)

• install.sh completed on a clean directory (venv, source-built empty-target vLLM, plugin wheel).
• run.sh booted to Application startup complete. Memory after prefill warmup: 30.47 of 31.58 GiB allocated, 1.11 GiB free.
• A 5-prompt coding probe gave 43.5 tok/s per user with 1,148 output tokens, identical to the same probe on the source-tree run.
• Passkey retrieval on the packaged server (one trial each):

prompt tokens	passkey retrieved	wall (prefill + 24 tokens)
3,062	yes	2 s
11,323	yes	11 s
14,364	yes	8 s

• Run on chip 2 of a p300c board, selected through a 2-chip lease (the bundle uses the first granted chip). The first boot took about 28 minutes because kernels were compiled cold while a benchmark shared the machine.
• Not verified: a fresh tt-model pull from the Hub, starting with an empty Hugging Face cache (drafter download), and any host other than the one above.

Performance

Measured on the equivalent source-tree build of the same code (the wheels in this bundle were built from that tree), not on this packaged bundle. vllm bench serve, /v1/completions, random prompts of exactly ISL tokens, --ignore-eos, temperature 0; t/s aggregate output tokens per second, (t/s/u) per user. Means of 2-8 requests; - = pool cannot seat that many.

ISL / OSL	1 user	2 users
128 / 128	20.1 (20.1)	29.7 (14.8)
1,024 / 128	28.5 (28.5)	40.0 (20.0)
2,048 / 128	24.3 (24.3)	37.2 (18.6)
4,096 / 128	19.6 (19.6)	26.9 (13.4)
8,192 / 128	14.4 (14.4)	-
12,288 / 128	12.9 (12.9)	-
128 / 1,024	26.3 (26.3)	39.8 (19.9)
4,096 / 1,024	36.3 (36.3)	57.9 (29.0)

TTFT ms / TPOT ms:

ISL / OSL	1 user TTFT ms / TPOT ms	2 users TTFT ms / TPOT ms
128 / 128	234 / 48.4	349 / 65.2
1,024 / 128	770 / 29.3	1,153 / 41.3
2,048 / 128	1,049 / 33.3	1,570 / 41.8
4,096 / 128	2,098 / 34.9	3,141 / 50.3
8,192 / 128	4,252 / 36.7	-
12,288 / 128	6,484 / 26.9	-
128 / 1,024	233 / 37.8	350 / 50.0
4,096 / 1,024	2,098 / 25.5	3,141 / 31.5

SPEED-Bench coding (80 single-turn prompts, so this measures decode speed, not agentic use; greedy, thinking off, 2,048-token cap; evaluation-license dataset, not redistributed):

users	per-user tok/s	mean TPOT	aggregate t/s	mean TTFT	hit cap
1	39.4	25.41 ms	34.7	296 ms	5
2	30.1	33.2 ms	53.3	1033 ms	5

Plain single-chip decode on the same hardware runs at about 14 tok/s per user (short prompts). HumanEval pass@1 (164 problems, greedy, thinking off, sandbox with no network): 150/164 = 91.5% (4 hit the 1,024-token cap); plain 2-chip decode on the same harness scored 150/164. A single greedy sample per problem on easy problems: a sanity check, not a quality evaluation.

Limits and caveats

• Context is 16,384 tokens, not the model's 262,144, and that rules out agentic coding tools (see above). Everything that is not weights had to shrink to fit 32 GB: a 16.5K-token KV pool, a 256 MB trace region and a 2,048-token prefill chunk. A 32K-token pool booted but ran out of DRAM on a 7K-token prompt.
• Requires a code change that is not upstream. The serving class refuses any device count except 2 and 4. patches/0001-qwen36-dflash-allow-tp1-composite-verify.patch (on tt-metal 9f6b02d2d779) allows one device with the composite verify only. The tt-metal-models wheel here includes it.
• Greedy only; at most 2 users (48 value heads per chip leaves a core budget for 2). Output is not byte-identical to plain greedy decoding.
• Needs TT_METAL_PINNED_MEMORY_CACHE_LIMIT_BYTES=0 (set in run.sh): without it weight load hangs on the test box, in this tt-metal build.
• run.sh was edited after generation to add the fixed vLLM arguments (--additional-config, prefill chunk, parsers, --no-async-scheduling), because package-thin has no flag for them.
• Selecting two chips of the second board through TT_VISIBLE_DEVICES failed in ttnn.get_num_devices() (unordered_map::at) on the test box, for this build and for a source build; single-chip selection worked.

Provenance

tt-metal	tenstorrent/tt-metal branch qwen36-p150x2 at 9f6b02d2d779 (2026-09-29) plus the patch above; wheels versioned 0.79.0.dev20260929+qwen36p150x2.g9f6b02d
Plugin	changh95/vllm-tt-plugin ab5f7f3d795e (Apache-2.0)
vLLM	0.26.0, built with VLLM_TARGET_DEVICE=empty at install
Python	3.12; transformers 5.17.0, tokenizers 0.23.2, CPU torch

Licenses

Base model and drafter: Apache-2.0. tt-metal and vllm-tt-plugin: Apache-2.0. This bundle redistributes built ttnn and tt-metal-models wheels from the tt-metal sources above.