--- license: gemma base_model: google/gemma-3-1b-it base_model_relation: quantized pipeline_tag: text-generation library_name: litert-lm tags: - litert-lm - litertlm - gemma3 - chat - text-generation - npu - edge-tpu - tensor-g4 - pixel - on-device - quantized - int4 extra_gated_heading: Access Gemma3-1B-IT (Tensor G4 NPU) on Hugging Face extra_gated_prompt: >- These are quantized, repackaged derivatives of google/gemma-3-1b-it and are governed by the Gemma license. To access them you must review and agree to Google's Gemma Terms of Use. Please ensure you are logged in to Hugging Face and click below. extra_gated_button_content: Acknowledge license --- # Gemma3-1B-IT — Tensor G4 NPU (AOT, int4 per-channel + Hadamard) > [!NOTE] > A quantized, repackaged variant of > [`google/gemma-3-1b-it`](https://huggingface.co/google/gemma-3-1b-it) that runs on the > **Google Tensor G4 Edge TPU ("rio") NPU** — the SoC in the Pixel 9 / 9 Pro / 9 Pro XL / 9 Pro Fold. > Packaged as a `.litertlm` bundle for the > [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. This is, to our knowledge, the **first publicly available int4 AOT Tensor-G4 NPU build of Gemma 3 1B** — and it is **~half the size of the only other Tensor-NPU Gemma 3 1B bundle** published (`litert-community/Gemma3-1B-IT_q8_ekv1280_Google_Tensor_G5.litertlm` is **1,678 MB**; this build is **875 MB = 0.52×**), because it is int4-per-channel rather than int8. `litert-community/Gemma3-1B-IT` ships Tensor **G5** and **G6** builds. There is no G4 build there. --- ## The build | | | |---|---| | File | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm` | | Mode | **AOT** — precompiled DarwiNN DGC | | Quantization | int4 **per-channel + Hadamard rotation** (all FC / BMM / embed) | | NPU share | **~100%** | | Embedded KV cache | 4096 | | Size | 874,631,968 B (875 MB) | 26-layer Gemma 3 1B-IT, embedded KV cache, embedded tokenizer + chat template, `.litertlm` v1. Built from Google's mobile-QAT checkpoint [`google/gemma-3-1b-it-qat-int4-unquantized`](https://huggingface.co/google/gemma-3-1b-it-qat-int4-unquantized). --- ## Performance (Pixel 9 Pro XL, Tensor G4) Measured on this exact bundle, 2026-09-10: | prefill | decode | prefill tok/s | decode tok/s | |---|---|---|---| | 128 | 64 | 608.83 | 15.93 | | 512 | 128 | 659.22 | 16.17 | | 1024 | 64 | 663.99 | 15.89 | **Prefill scales; decode is pinned.** Prefill rises 608 → 659 → 664 tok/s as the batch grows 128 → 1024, while decode holds at 15.89–16.17 tok/s regardless (±0.9% across an 8× change in prefill length). That is a clean confirmation that decode is **dispatch-bound, not compute-bound** — more work per dispatch makes prefill faster, but decode always does exactly one token per dispatch, so nothing moves it. The rise from 608 to 664 is fixed per-dispatch overhead amortizing: at 128 tokens you pay it once per 128, at 1024 once per 1024. Extrapolating, the per-dispatch prefill floor is roughly **~670 tok/s** for this model. **AOT profile = prefill-monster / decode-laggard.** The Edge TPU crushes batched matmuls (prefill) but a single-token decode underutilizes it — so this build is the fastest option on the phone for prefill-heavy work (RAG, long-context, summarization, classification), and a strong speculative-decode (MTP) candidate. Speculative decode is the real next perf lever and is not wired into this build. > Behaviour/"feel" differences vs a reference are **runtime config** (system prompt + sampling), not the > quant — a faithful int4 copy matches the reference once system prompt and top-k / top-p / temperature match. --- ## Quantization ### The rank-2 FC surgery — the key enabler (affects G4 **and** G5) The DarwiNN AOT compiler `OP_FAILED`s on **rank-3 `FULLY_CONNECTED`** ops, which Gemma-3 emits ~360× — so the AOT compile fails outright without a fix. **`recipe/fc_rank2_surgery.py`** wraps every rank-3 FC as `RESHAPE → (rank-2) FC → RESHAPE` (same math, a rank the compiler accepts). This is what lets a full int4 Gemma 3 actually compile to a single G4 DGC. We are not aware of an upstream fix. Details in **[`RANK2_FC_SURGERY.md`](./RANK2_FC_SURGERY.md)**. ### Hadamard rotation On top of int4 per-channel, this build applies a **Hadamard rotation** to the weights before quantization (a `DECOMPOSED_HADAMARD_ROTATION`-style outlier-suppression transform). Spreading outliers across the rotated basis tightens the per-channel int4 grid — most visibly on the 262k-vocab head, where it improves logit precision and coherence over plain per-channel int4, at a small size cost. ### Per-quant AOT support is compiler-build-dependent There is **no fixed per-bit-width support matrix** for the DarwiNN AOT compiler — what a given toolchain build accepts varies by compiler version. The shipping `gemma4-E2B` G4-NPU bundle is a **2-subgraph** model carrying **int2, int4, and int8** weight tensors (verified directly), so mixed int2/int4/int8 does AOT-compile. The verified enabler for compiling a full Gemma to a G4 DGC is the **rank-2 FC surgery**, not any particular bit width. Granularity is a build choice, not a compiler wall: - **int4 per-channel + Hadamard** (one scale per output channel, rotated) → this build. - **int4 per-channel** (one scale per output channel) → also coherent. - **int4 per-tensor** also AOT-compiles, but one scale over the 262k-vocab head → word salad. --- ## Running This is a **precompiled DarwiNN DGC**. It requires a **compatible GoogleTensor dispatch library** to load on-device; a stock app's dispatch may reject it as a firmware mismatch. Runtime/dispatch specifics are out of scope for this repo. | Surface | Support | |---|---| | LiteRT-LM CLI (`--backend=npu`) | ✅ with a compatible dispatch runtime | | Google AI Edge Gallery (NPU) | needs a compatible dispatch runtime | | Tensor G4 (Pixel 9 series) | ✅ | | Tensor G5 (Pixel 10) | ❌ DGC is G4-targeted | | Older Tensor (G2 / G3) | ❌ | | llama.cpp / GGUF, Web, iOS, desktop, non-Pixel | ❌ `.litertlm` is Pixel-NPU-only | --- ## Files | File | Purpose | |---|---| | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm` | the model — per-channel int4 + Hadamard, 4096 ctx, ~100% NPU | | `RANK2_FC_SURGERY.md` | the rank-2 FC surgery writeup + why per-quant AOT support is compiler-build-dependent | | `recipe/fc_rank2_surgery.py` | rank-3 → rank-2 FC surgery (the DarwiNN AOT enabler) | | `MANIFEST.md` | checksum + size | --- ## License & attribution Base model **Gemma 3 1B-IT** © Google, under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). This quantized/repackaged derivative inherits the Gemma license. Runtime: **LiteRT-LM** / **LiteRT** (Apache-2.0). Quantization, rank-2 FC surgery, per-channel int4 + Hadamard AOT compile, and G4 NPU packaging by **@xThr45hx**.