gemma-4-26b-a4b-it on Tenstorrent p150 and p150x2 and p150x4
Serves google/gemma-4-26B-A4B-it by Google DeepMind at revision 4d7ae498. See the original model card for training and evaluation details.
Gemma 4 26B A4B instruction-tuned, a 25.2B-parameter mixture-of-experts model (3.8B active parameters; 30 layers; 128 experts with 8 active plus 1 shared), served with vLLM for chat, reasoning and tool calling on text input.
Runs on Tenstorrent p150 or p150x2 or p150x4 — see the serving profiles below.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
At a glance
| Hardware | p150, p150x2, p150x4 |
| Context | 262,144 tokens |
| License | apache-2.0 |
Intended use
Chat, reasoning and tool calling on text through an OpenAI-compatible API.
Prerequisites
- The Tenstorrent CLI,
tt:uv tool install tenstorrent(or use tt-model alone, see the Quickstart) - Docker
- Tenstorrent hardware: p150 or p150x2 or p150x4
Quickstart
tt serve tt-hous/gemma-4-26b-a4b-it
tt serve downloads the Docker image and the google/gemma-4-26B-A4B-it at 4d7ae4984b7db7de8f8457170b3f1a419ee76d52 weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.
Without tt-cli: tt-model serve tt-hous/gemma-4-26b-a4b-it.
Serving profiles
Select a different serving profile using --profile.
| profile | hardware | mesh | max_num_seqs | max_model_len | description |
|---|---|---|---|---|---|
p150 |
p150 | P150 | 32 | 131072 | Single P150; 131,072-token context. The KV pool is sized from the DRAM left after the weights load (1,509 blocks; one 131k request or about 16 concurrent 1k-context sequences), with an admission reserve so residents are not preempted. Prefill attention runs 128-row query and key chunks on this profile. |
p150x2 (default) |
p150x2 | P150x2 | 32 | 262144 | Two chips, (1, 2) mesh, FABRIC_2D; 262,144-token context. Validated on a TT-QuietBox 2 (2x P300C, 4 chips), where the launch opens a 2x2 parent mesh and serves on the (1, 2) submesh at offset (0, 0). |
p150x4 |
p150x4 | P150x4 | 32 | 262144 | Four P150 boards, (1, 4) mesh, FABRIC_1D_RING; 262,144-token context. |
Capabilities
The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "google/gemma-4-26B-A4B-it" — the weights id, not this package's name.
Tool calling is enabled (--tool-call-parser gemma4): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.
Reasoning output is separated (--reasoning_parser gemma4): the thinking text arrives in reasoning (reasoning_content on vLLM before 0.11.1), apart from content.
Automatic prefix caching is on: a request that extends an earlier one prefills only its new
tokens. On p150, a 16,384-token shared prefix took 24.8 s to first token cold and 179 ms
cached.
The device sampler covers top_k up to 64, so the checkpoint's default recipe
(temperature 1.0, top_p 0.95, top_k 64, applied when a request sets no sampling fields)
runs at the greedy rate. Requests with top_k above 64, a seed, penalties or logprobs
are sampled on the host, about 2.3 times slower per token at one user.
Expected performance
| eval | this package | reference | prompts | notes |
|---|---|---|---|---|
| GPQA Diamond | 81.31% | 82.3% | 198 | p150x4; thinking on, temperature 1.0, top_p 0.95, top_k 64, 98,304 output tokens; standard error 2.77 |
| IFEval, prompt-level strict / loose | 88.17% / 89.65% | — | 541 | p150; greedy, 1,280 output tokens |
| IFEval, instruction-level strict / loose | 91.61% / 92.81% | — | 541 | same run |
| GPQA (main set), CoT zero-shot | 64.06% | — | 448 | p150; greedy, seed 42, 2,048 output tokens; standard error 2.27 |
| SWE-bench Verified | 1 of 5 resolved | — | 5 tasks | p150x2; mini-swe-agent, 250 steps, 32,768 output tokens |
| Terminal-Bench 2 | 2 of 5 resolved | — | 5 tasks | p150x2; harbor, same recipe |
| Guided JSON output | 100 of 100 valid | — | 100 | p150x2; greedy |
Evals ran through tt-inference-server's eval workflow, 0-shot. The reference is Google's figure from the original model card, under Google's own harness; the Gemma 4 card publishes no IFEval, GPQA main-set, SWE-bench Verified, Terminal-Bench 2 or guided-JSON figure. GPQA Diamond ran from 2026 Sep 25 to 29, the GPQA main set on Oct 8, IFEval on Oct 10, the agentic and JSON rows earlier; all on builds of this package that precede the published image (built 2026 Oct 13).
| profile | ISL | OSL | concurrency | N | TTFT (ms) | prefill tok/s/u | decode tok/s/u | E2EL (s) |
|---|---|---|---|---|---|---|---|---|
| p150 | 128 | 128 | 1 | 1 | 175 | 731 | 36.9 | 3.62 |
| p150 | 128 | 128 | 8 | 1 | 1,191 | 107 | 14.1 | 10.31 |
| p150 | 128 | 128 | 32 | 1 | 3,921 | 33 | 11.3 | 15.32 |
| p150 | 1,024 | 256 | 1 | 1 | 1,101 | 930 | 35.9 | 8.20 |
| p150 | 1,024 | 256 | 8 | 1 | 5,351 | 191 | 10.3 | 30.53 |
| p150 | 1,024 | 256 | 32 | 1 | 48,481 | 21 | 9.8 | 75.18 |
| p150 | 4,096 | 256 | 1 | 1 | 4,375 | 936 | 35.2 | 11.62 |
| p150 | 4,096 | 256 | 8 | 1 | 21,122 | 194 | 8.1 | 55.60 |
| p150 | 4,096 | 256 | 32 | 1 | 157,102 | 26 | 4.0 | 223.93 |
| p150 | 16,384 | 256 | 1 | 1 | 18,006 | 910 | 34.4 | 25.42 |
| p150 | 16,384 | 256 | 8 | 1 | 88,687 | 185 | 4.3 | 158.87 |
| p150 | 32,768 | 256 | 1 | 1 | 37,205 | 881 | 33.3 | 44.85 |
| p150 | 65,536 | 256 | 1 | 1 | 78,899 | 831 | 31.5 | 86.99 |
| p150x2 | 128 | 128 | 1 | 1 | 167 | 766 | 46.2 | 2.92 |
| p150x2 | 128 | 128 | 8 | 1 | 1,146 | 112 | 14.9 | 9.71 |
| p150x2 | 128 | 128 | 32 | 1 | 3,706 | 35 | 13.7 | 13.13 |
| p150x2 | 1,024 | 256 | 1 | 1 | 1,100 | 931 | 44.5 | 6.83 |
| p150x2 | 1,024 | 256 | 8 | 1 | 5,378 | 190 | 11.8 | 27.20 |
| p150x2 | 1,024 | 256 | 32 | 1 | 18,645 | 55 | 8.2 | 52.97 |
| p150x2 | 4,096 | 256 | 1 | 1 | 4,464 | 918 | 43.3 | 10.36 |
| p150x2 | 4,096 | 256 | 8 | 1 | 23,565 | 174 | 6.0 | 71.84 |
| p150x2 | 4,096 | 256 | 32 | 1 | 63,451 | 65 | 3.2 | 160.23 |
| p150x2 | 16,384 | 256 | 1 | 1 | 18,401 | 890 | 42.7 | 24.37 |
| p150x2 | 16,384 | 256 | 8 | 1 | 82,603 | 198 | 4.5 | 166.84 |
| p150x2 | 32,768 | 256 | 1 | 1 | 39,345 | 833 | 42.1 | 45.40 |
| p150x2 | 65,536 | 256 | 1 | 1 | 84,812 | 773 | 40.6 | 91.09 |
| p150x2 | 131,072 | 256 | 1 | 1 | 198,144 | 661 | 37.6 | 204.93 |
| p150x4 | 128 | 128 | 1 | 1 | 166 | 771 | 49.3 | 2.74 |
| p150x4 | 128 | 128 | 8 | 1 | 1,099 | 116 | 19.4 | 7.68 |
| p150x4 | 128 | 128 | 32 | 1 | 3,638 | 35 | 14.8 | 12.42 |
| p150x4 | 1,024 | 256 | 1 | 1 | 1,062 | 964 | 47.6 | 6.42 |
| p150x4 | 1,024 | 256 | 8 | 1 | 5,190 | 197 | 15.7 | 21.78 |
| p150x4 | 1,024 | 256 | 32 | 1 | 17,982 | 57 | 9.2 | 49.07 |
| p150x4 | 4,096 | 256 | 1 | 1 | 4,302 | 952 | 46.1 | 9.83 |
| p150x4 | 4,096 | 256 | 8 | 1 | 19,325 | 212 | 10.4 | 47.68 |
| p150x4 | 4,096 | 256 | 32 | 1 | 61,298 | 67 | 3.4 | 153.38 |
| p150x4 | 16,384 | 256 | 1 | 1 | 17,711 | 925 | 45.0 | 23.38 |
| p150x4 | 16,384 | 256 | 8 | 1 | 79,824 | 205 | 5.4 | 156.69 |
| p150x4 | 32,768 | 256 | 1 | 1 | 36,270 | 903 | 44.5 | 42.00 |
| p150x4 | 32,768 | 256 | 8 | 1 | 165,936 | 197 | 3.7 | 310.55 |
| p150x4 | 65,536 | 256 | 1 | 1 | 76,968 | 851 | 42.0 | 83.03 |
| p150x4 | 131,072 | 256 | 1 | 1 | 169,011 | 776 | 39.5 | 175.46 |
Means over one run per row on a QuietBox 2 (two P300C boards), with 1, 2 or 4 chips opened
for p150, p150x2 and p150x4, served with tt-model serve: random-token prompts of
exactly ISL tokens, output pinned to OSL tokens with ignore_eos, greedy, one warmup request
per input length, 4 requests per user up to 4,096 input tokens, 2 per user up to 16,384 and
1 per user above. The rows are selected from a sweep at 1 to 32 concurrent users; rows
predicted to exceed 300 s end to end were skipped. The p150 rows were measured on
2026 Oct 10, the p150x2 and p150x4 rows on 2026 Sep 24, on builds of this package that
precede the published image (built 2026 Oct 13). Prefill tok/s/u is ISL divided by TTFT, so above concurrency 1
it includes queueing time.
Limitations
- Text input only. Tested on one QuietBox 2 only; separate P150 boards were not tested.
- Evals and speed were measured on builds of this package that precede the published image, from 2026 Sep 24 to Oct 10, and were not re-run on the published image.
- Not for long-context interactive use at high concurrency: at 16,384 input tokens and 8 users, a request takes about 160 s end to end on every profile.
- Some long generations enter a repetition cycle and never emit an answer: 1 of 90 GPQA prompts at a 2,048-token budget, and 18 of 198 GPQA Diamond responses ran to the 98,304-token budget without an answer.
- Guided JSON with an optional UUID field hit a 128-token budget mid-field in 100 of 100 requests. Give structured output a budget that fits the full object.
Feedback
Questions or problems with this package: open a discussion at https://huggingface.co/tt-hous/gemma-4-26b-a4b-it/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: [email protected].
Provenance
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | e5a5653720f1679517dd05ab01f66c4e036aa515 (dirty tree — the image includes uncommitted changes) |
| vLLM | vllm-0.1.dev14200+gc87df030e.empty.gemma4.empty-py3-none-any.whl — a wheel the author built |
| vllm-tt-plugin | a local checkout — commit not published |
code/ digest |
03d83b851962b859 (sha256, first 16 hex digits) |
| built | 2026-10-13T17:30:56+00:00 by tt-model 0.1.0 |