IBM Granite 4.1-8B on Tenstorrent tt-metal (community port)
⚠️ 重要: このリポジトリは Model Card のみです — weights はありません
⚠️ Card-only repository — NO WEIGHTS ARE HOSTED HERE
この Hugging Face リポジトリは 互換性情報を掲載するためのミラー であり、
そのまま読み込めるモデルリポジトリではありません。
.safetensors / .bin / .gguf などのモデルファイルはなく、公開されているのは
この Model Card (README) のみです。unnowataru/granite-4.1-8b-tt-metal を
from_pretrained()、vLLM、Hugging Face Inference API、TGI、その他の一般的な
モデルローダーに指定しても動作しません。
実際の weights は、IBM の公式リポジトリから取得してください:
ibm-granite/granite-4.1-8b
Tenstorrent Wormhole 上で実行するためのポート実装、検証スクリプト、Quickstart は
GitHub にあります:
unnowataru/granite-4.1-tt-metal
This HuggingFace repository is a compatibility mirror, not a loadable model. There is no
.safetensors / .bin / .gguf here — this repo is just this README.
# ❌ These will all FAIL — this repository contains no model artifact:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("unnowataru/granite-4.1-8b-tt-metal")
from vllm import LLM
llm = LLM(model="unnowataru/granite-4.1-8b-tt-metal")
# Any HF Inference API / TGI / generic loader with this repo ID will also fail.
Use the upstream IBM repository instead for the actual weights:
# ✅ Correct: weights come from IBM's Apache-2.0 repository
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"ibm-granite/granite-4.1-8b",
revision="1504002f650e656a0a3789d99574df12e3e94ed0" # pinned by this port
)
For running these weights on Tenstorrent Wormhole hardware, see the port implementation:
unnowataru/granite-4.1-tt-metal on GitHub.
Start there with scripts/quickstart.sh.
library_name, pipeline_tag, and base_model are intentionally omitted from this repo's
metadata because it contains no loadable model artifact and no derivative weights.
非公式・非提携のコミュニティポートです。 本プロジェクトは IBM、Tenstorrent、株式会社ネットワールド (Networld Corporation) の いずれとも提携・関連関係になく、各社は本プロジェクトの開発、検証、公開に関与していません。 また、本プロジェクトは各社によって承認または推奨されたものではありません。
Community, non-affiliated port. IBM, Tenstorrent, and Networld Corporation were not involved in the development, validation, or publication of this project and do not endorse it.
What this repository provides
This HuggingFace repository is a model card describing a community port of
ibm-granite/granite-4.1-8b to Tenstorrent
Wormhole hardware (via tt-metal and the
Tenstorrent vLLM plugin).
The weights and implementation live elsewhere:
- Weights: obtain from IBM's
ibm-granite/granite-4.1-8bdirectly (Apache-2.0). - Port implementation, verification harness, and reproduction scripts:
unnowataru/granite-4.1-tt-metalon GitHub.
Upstream model and ownership
- Model: IBM Granite 4.1-8B instruction-tuned model (the no-suffix repository; the
-baserepository denotes the base checkpoint). - Copyright: International Business Machines Corporation.
- License: Apache-2.0.
- Weights are not modified or redistributed by this repository. Download and use IBM's weights directly.
Supported hardware and software stack
All measurements used one pinned software/firmware stack but two device configurations: N150 for single-device numerical checks and N300 for serving and performance measurements. Cross-stack and independent cross-device reproduction have not been tested; other revisions or configurations may differ.
| Component | Version / pin |
|---|---|
| Tenstorrent card | Wormhole b0 |
| Device configurations (measured) | N150 for single-device numerical validation; N300 for serving and performance |
| Firmware bundle | 19.6.0 |
| KMD | 2.7.0 |
tt-metal |
commit 805f43d (specific main point, not a release tag) |
| Container image | ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.12.0-805f43d-a45c614 |
| Container image digest | sha256:6cbb9532aa7a74cac9805364a0f240694166ee291bf3cc04f8dd7d7a88d7d8a7 |
| Container Python | 3.10 |
| Upstream model revision | ibm-granite/granite-4.1-8b snapshot 1504002f650e656a0a3789d99574df12e3e94ed0 (pin your HF_REVISION to this SHA for exact reproduction) |
| Port revision | release tag v0.1.0 (see the GitHub repo's tags for the corresponding Git commit SHA) |
T3K (8-device) has not been measured.
Abbreviated integration path
This is a condensed integration outline. It starts after entering the pinned container
(see the GitHub repo's docs/stack.md
and the full reproduction procedure for the clean-host startup). For a clean-host reproduction
from scratch, follow the pinned full procedure in the GitHub repository rather than this excerpt.
# Assumes you are inside the pinned container (image digest above), with tt-metal at commit 805f43d.
export TT_METAL_HOME=/home/container_app_user/tt-metal
test "$(git -C "$TT_METAL_HOME" rev-parse --short HEAD)" = "805f43d" || {
echo "tt-metal at unexpected revision; abort"; exit 1
}
# Fetch the port at a specific release tag
git clone --branch v0.1.0 --depth 1 https://github.com/unnowataru/granite-4.1-tt-metal
cd granite-4.1-tt-metal
# Overlay the 6-file Granite diff onto the tt-metal tree
cp overlays/tt_transformers/tt/*.py "$TT_METAL_HOME/models/tt_transformers/tt/"
# Register TTGraniteForCausalLM in the vLLM(tt) plugin (idempotent)
python scripts/serving/patch_tt_py_granite.py
# Start the OpenAI-compatible server, pinning the HF model snapshot
export HF_MODEL=ibm-granite/granite-4.1-8b
export HF_REVISION=1504002f650e656a0a3789d99574df12e3e94ed0
export MESH_DEVICE=N300
bash scripts/serving/run_granite_server.sh
Note: the patch_tt_py_granite.py script targets the vLLM(tt) plugin bundled in the pinned
container image. If your plugin path or revision differs, the patcher should refuse to apply —
verify the run header before proceeding.
The full server setup, verification harness, and integration tests are documented in the GitHub repository.
Observed checks (scope-limited; on the stack above)
These are the specific checks performed on the pinned stack, not general capability claims.
- One full 40-layer teacher-forced forward pass completed on N150 (single-prompt, next-token logit comparison against HF CPU bfloat16 reference): PCC = 0.999410, top1 match, top5 overlap 4/5.
- One manual generation spot-check on N150 (approximately 280 prompt tokens, 130 generated tokens, greedy, EOS-terminated): the preserved output was judged semantically responsive and free of gibberish. This is not a long-context or model-quality evaluation.
- Serving on N300 via vLLM(tt): both
/v1/completionsand/v1/chat/completionsrespond; the chat_template round-trip between the two endpoints is byte-exact on the fixed test message. - One scripted structured-tool-calling fixture on N300 passed the following checks: the
function name, JSON-valid arguments, the expected argument value (
city=Tokyo), the final answer's reflection of the returned tool data, and the streaming form's reconstruction of a valid tool call. This uses--tool-call-parser hermes(parser name matters; seedocs/traps.md). - OpenAI Python SDK smoke tests passed for the following seven tested operations:
/v1/models, system+user, multi-turn context in a scripted 2-turn exchange, streaming, four concurrent requests,stopparameter, greedy determinism across two identical calls.
Not yet validated (open gaps, honestly disclosed)
- Multi-device teacher-forced numerical comparison on N300 / T3K. The single-device (N150)
numerical pass does not validate the tensor-parallel path — TP introduces
mesh_partition/all_gather/all_reducecollective operations that the harness did not cover. - Multi-prompt statistical numerical coverage. Only one prompt has been used for the teacher- forced comparison. A defensible statistical claim requires many more independent trials.
- Autoregressive long-generation agreement. Autoregressive token-by-token agreement over hundreds to thousands of generated tokens has not been measured. The teacher-forced next-token- logit comparison does not establish long-horizon greedy-decoding agreement.
- Chat/instruct decoding path in the numerical harness. The generation spot-check ran with
instruct=0(base decoding); the chat/instruct decoding path is only exercised by the serving smoke, not by the numerical harness. max_num_seqs = 8on N300. One attempt ended with a chip-lock error consistent with a stale lock from a prior process, before model capacity could be tested. This run provides no evidence either for or against N300 KV-cache capacity at msq=8; a clean-start measurement is still required.- Any T3K measurement. Not attempted in this session.
- Percentile latency (p50 / p95 / p99). Only means are reported below.
- Sustained-load characteristics. Runs are on the order of minutes, not hours.
Performance (measured 2026-07-11, N300 only)
Configuration abbreviations: msq = max_num_seqs, mml = max_model_len.
Method: Python requests + streaming SSE from a Windows client to the container's docker bridge
IP on port 8000 via SSH tunnel; one warm-up request (16 tokens) per configuration. Sample sizes
range from 1 (single-observation concurrency runs) to 5 (baseline single-stream) per data point —
each row lists its own n below. Values are arithmetic means of that row's sample; variance and
percentiles are not analyzed.
Single-stream
| Config | TTFT (s) | TPOT (ms/tok) | tok/s | n |
|---|---|---|---|---|
| N300, msq=2, mml=4096 | 0.126 | 42.7 | 23.5 | 5 |
| N300, msq=4, mml=8192 | 0.117 | 40.7 | 24.8 | 2 |
Concurrent (N300)
| Config | Concurrency | Aggregate tok/s | n | Notes |
|---|---|---|---|---|
| msq=2, mml=4096 | 2 | 41.3 | 1 | 1.76× single-stream aggregation observed |
| msq=2, mml=4096 | 4 (queue) | 44.4 | 1 | Two streams served immediately (~150 ms TTFT), two queued ~4.5 s |
| msq=4, mml=8192 | 3 (Hermes-agent delegation fit) | 46.0 | 1 |
Prefill (msq=2 mml=4096, greedy /v1/completions)
| Prompt tokens | TTFT_avg (s) | TPOT_avg (ms/tok) | n |
|---|---|---|---|
| 64 | 0.66 | 31.9 | 3 |
| 512 | 0.85 | 33.0 | 3 |
| 1024 | 0.85 | 33.2 | 3 |
| 2048 | 1.14 | 33.1 | 3 |
| 3800 | 1.91 | 34.0 | 3 |
Across these sample means, TPOT ranged from 31.9 to 34.0 ms/token while TTFT increased at longer prompt lengths. With only 3 runs per point and no variance analysis, these observations are descriptive; prompt-length independence of TPOT and a causal prefill attribution to the TTFT trend are not established.
Detailed methodology, raw evidence logs, and additional measurements: see
docs/performance.md
and the raw logs under docs/evidence/.
Kernel-imposed constraint worth knowing
The upstream tt_transformers model_config enforces supported_batches = {1, 2, 4, 8, 16, 32}.
max_num_seqs = 3, 5–7, etc. fail at startup with
ValueError: Batch size N not supported. No fine-grained tuning between the powers of two.
Reproduction
Every measured number in this card is tied to the revisions listed in the Supported stack table
above; the linked GitHub release contains the corresponding harness scripts and raw logs. The
full reproduction procedure — clean-host startup, dev-container launch, overlay + registration
patch, weight download with HF_REVISION pinning, server startup, smoke — is documented step-by-
step in the GitHub repository. See
docs/validation.md
and docs/performance.md
for links to each raw log.
Known limitations (in addition to Not yet validated above)
- Single stack. All numbers anchor to the pinned software/firmware stack. Drift on any
component is uncharacterized. Upstream
tt-metalmoves rapidly. - No SLA. This is a community port, single maintainer. Response times on issues are best-effort.
- No re-hosted weights. Users must obtain the weights separately and comply with their applicable license; this repository neither redistributes the weights nor grants additional rights in them.
License, attribution, and non-affiliation
- This repository's content: Apache-2.0.
- Model weights: IBM's, Apache-2.0, not redistributed here.
- Overlays under
overlays/tt_transformers/tt/: Derivative Works of upstreamtt_transformersattt-metalcommit805f43d. Each file carries aSPDX-License-Identifier: Apache-2.0header and a "Modified for Granite 4.1" notice per Apache-2.0 §4(b). - See the GitHub repository's
NOTICEfor the full attribution list.
Non-affiliation and trademarks: this project is not endorsed by, or officially affiliated with, International Business Machines Corporation, Tenstorrent Inc., or Networld Corporation. Names such as IBM, Granite, Tenstorrent, Wormhole, and tt-metal are trademarks or product names of their respective owners. These names are used only to identify the upstream components; no trademark license, endorsement, or affiliation is claimed.
Citation
If you use this port in academic or industry work, please cite both the upstream Granite model and this repository:
IBM Granite 4.1-8B — https://huggingface.co/ibm-granite/granite-4.1-8b
granite-4.1-tt-metal (community port) — https://github.com/unnowataru/granite-4.1-tt-metal
release v0.1.0