litert-community/Gemma3-1B-IT-Tensor-NPU

Gemma3-1B-IT — Google Tensor NPU (AOT, int4 per-channel + Hadamard)

Quantized, repackaged variants of google/gemma-3-1b-it precompiled (AOT) for the Google Tensor Edge TPU NPU across Pixel generations. Packaged as .litertlm bundles for LiteRT-LM. (This repo was previously named Gemma3-1B-IT-Tensor-G4-NPU.)

Community build — not an official Google release. Built and maintained by @xThr45hx; not affiliated with or endorsed by Google.

Variants

SoC	Pixel	File	Status
Tensor G4	Pixel 9 / 9 Pro / 9 Pro XL / 9 Pro Fold	Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm	✅ VERIFIED on device
Tensor G3	Pixel 8 / 8 Pro / 8a	Gemma3-1B-IT_int4hadamard_aot_ekv4096_G3.litertlm	🟡 Same program as the verified G4 build; replies word-for-word identically to the G4 build on G4 hardware — awaiting a Pixel 8 report
Tensor G5	Pixel 10 series	Gemma3-1B-IT_int4hadamard_aot_ekv4096_G5.litertlm	⚠️ UNVERIFIED on hardware (compiled cleanly via Beta AOT, authentic buenos_b0 DGC0, awaiting Pixel 10 hardware)
Tensor G6	Pixel 11 series	Gemma3-1B-IT_int4hadamard_aot_ekv4096_G6.litertlm	❌ Tested on physical Pixel 11 Pro XL (kodiak) — Fails inference warmup (\Failed to invoke the compiled model). Preview \santafe_a0\ AOT compile is incompatible with retail G6 silicon/firmware. Use the JIT / regular pipeline.

Tensor G6 physical test result (Pixel 11 Pro XL, kodiak): Tested on physical Pixel 11 Pro XL (kodiak, Android 16, kernel 6.12.69). The AOT bundle compiled via the Beta SDK santafe_a0 target fails during engine creation warmup with: RET_CHECK failure (llm_litert_npu_compiled_model_executor.cc:434) result Inference warmup run for Text Decoder (prefill) failed. Failed to invoke the compiled model. The preview AOT microcode registers/ABIs do not match retail G6 silicon firmware. Tensor G6 users should use uncompiled JIT / regular pipeline models instead, which run smoothly and rapidly directly on the device NPU.

[!WARNING] Tensor G5 build is UNVERIFIED on physical hardware. Compiled cleanly using the LiteRT Beta SDK with LMS=True (bare) and full DarwiNN DGC0 RISC-V codegen (uenos_b0 for G5). Because physical Tensor G5 devices are not yet available, this build is provided as an experimental preview.

Tensor G3 is not yet confirmed on a Pixel 8. Strong evidence it works: the G3 and G4 compiles are the same program. They differ in only 19 bytes (a device-type tag and its hash), and the G3 bundle runs on a Pixel 9 (G4) and gives word-for-word identical replies to the G4 build. If you have a Pixel 8 / 8 Pro / 8a, please test it and comment on this repo (Community tab) with: device, whether it loads on the NPU, tokens/s, and a sample multi-sentence reply. That's what moves it to ✅.

Cross-generation DGC compatibility (tested on a Pixel 9 Pro XL / Tensor G4)

• G3 ↔ G4: the G3 and G4 builds compile to the same program. They differ only in a device-type field (03 vs 04) in the embedded compiler options, plus its 8-byte hash. The G3 bundle loads and gives word-for-word identical replies to the G4 build on G4 hardware, so the G4 runtime does not enforce the tag.
• G5 → G4: in an earlier test, a G5-compiled Gemma4-E2B loaded on G4 and its prefill executed on the G4 NPU, but decode did not. G5 compiles use dual-cluster optimizations that G4 hardware lacks, so newer-generation DGCs are only partly forward-loadable.
• Takeaway: use the build for your chip. G3 and G4 share a program; G5 and G6 need their own compiles.

All variants: 26-layer Gemma 3 1B-IT, 4096 context (embedded KV cache), prefill 128, embedded tokenizer + chat template, .litertlm v1, ~875 MB, built from Google's QAT checkpoint google/gemma-3-1b-it-qat-int4-unquantized.

This was, to our knowledge, the first publicly available int4 AOT Tensor-NPU build of Gemma 3 1B, at ~half the size of the int8 Tensor-NPU Gemma 3 1B bundles (e.g. the 1,678 MB G5 int8 build): 875 MB = 0.52×.

---

⚠️ How to run it — use Box, not stock Edge Gallery

These are precompiled DarwiNN DGCs. They need a compatible GoogleTensor dispatch runtime on the device. Stock Google AI Edge Gallery will not run them (its runtime/dispatch rejects them).

👉 Recommended: Box — ships the correct runtimes/dispatch libs. Download the APK, put the .litertlm for your SoC on the phone, load it with the NPU backend.

Surface	Support
Box (github.com/jegly/Box)	✅ recommended
LiteRT-LM CLI (--backend=npu)	✅ with a compatible dispatch runtime
Stock Google AI Edge Gallery	❌ wrong dispatch runtime
Use the file for your SoC only	a DGC compiled for one Tensor generation won't run on another
llama.cpp / GGUF, Web, iOS, desktop, non-Pixel	❌ .litertlm DGCs are Pixel-NPU-only

---

Performance (Pixel 9 Pro XL, Tensor G4)

Measured 2026-09-10 on the G4 bundle:

prefill	decode	prefill tok/s	decode tok/s
128	64	608.83	15.93
512	128	659.22	16.17
1024	64	663.99	15.89

Prefill scales, decode is pinned (~16 tok/s, dispatch-bound): fastest option on the phone for prefill-heavy work (RAG, long context, summarization, classification).

---

The exact AOT recipe (reproducible)

Full step-by-step in AOT_RECIPE.md; executable version with guardrails in recipe/gemma3_int4hr_recipe.py. In short:

1. Source: QAT weights google/gemma-3-1b-it-qat-int4-unquantized (not plain -it).
2. Export: litert_torch export_hf — split_cache=True, cache_length=4096, prefill_lengths=[128], externalize_embedder=True. Use the float prefill_decode (model.tflite), not the default int8 copy export_hf also writes.
3. Quantize: ai_edge_quantizer dynamic_wi4c_hr_afp32 — int4 per-channel + Hadamard rotation, applied once.
4. Rank-2 FC surgery: recipe/fc_rank2_surgery.py (see RANK2_FC_SURGERY.md). Used for G4 and G3.
5. Compile: aot_compile(target=[Tensor_Gx], google_tensor_enable_large_model_support=True) — LMS only, no sharding flag.
6. Toolchain: ai_edge_litert 2.2.0 + the Google Tensor beta compiler (BuildID 87ea807a…).

Reproducibility check: a from-scratch rebuild with this recipe produces a compiled prefill_decode that is bit-identical to the published G4 build (md5 f1102df2…), and gives word-for-word identical replies on device.

Quantization notes

• Hadamard rotation is the coherence fix. Plain per-channel int4 is fluent-but-wrong; the same int4 + Hadamard is correct. Also what lets the 4096-context graph compile.
• int4 per-tensor compiles but word-salads (one scale over the 262k-vocab head).
• Blockwise int4 compiles but is incoherent on the NPU AOT path — per-channel is the NPU-legal path.
• Per-quant support is compiler-build-dependent; retest against your own compiler instead of trusting a table.

---

Files

File	Purpose
Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm	Tensor G4 — ✅ verified
Gemma3-1B-IT_int4hadamard_aot_ekv4096_G3.litertlm	Tensor G3 — 🟡 same program as G4, awaiting Pixel 8 report
Gemma3-1B-IT_int4hadamard_aot_ekv4096_G5.litertlm	Tensor G5 — ⚠️ unverified on hardware (buenos_b0)
Gemma3-1B-IT_int4hadamard_aot_ekv4096_G6.litertlm	Tensor G6 — ⚠️ unverified on hardware (santafe_a0)
AOT_RECIPE.md	exact reproducible AOT recipe
recipe/gemma3_int4hr_recipe.py	executable recipe with toolchain/input guardrails
recipe/fc_rank2_surgery.py	rank-3 → rank-2 FC surgery
RANK2_FC_SURGERY.md	why the surgery is needed
MANIFEST.md	sizes + SHA-256

---

License & attribution

Base model Gemma 3 1B-IT © Google, under the Gemma Terms of Use. These quantized/repackaged derivatives inherit the Gemma license. Runtime: LiteRT-LM / LiteRT (Apache-2.0). Quantization, rank-2 FC surgery, per-channel int4 + Hadamard AOT compile, and Tensor NPU packaging by @xThr45hx. Not affiliated with or endorsed by Google.