Text Generation
LiteRT-LM
litertlm
gemma3
chat
npu
edge-tpu
tensor-npu
tensor-g3
tensor-g4
pixel
on-device
quantized
int4
Instructions to use litert-community/Gemma3-1B-IT-Tensor-NPU with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Gemma3-1B-IT-Tensor-NPU with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Gemma3-1B-IT-Tensor-NPU \ --prompt="Write me a poem"
- Notebooks
- Google Colab
- Kaggle
Docs: replace in-joke test prompt with md5 reproducibility check
Browse files- AOT_RECIPE.md +9 -8
- README.md +6 -5
AOT_RECIPE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# AOT recipe β Gemma3-1B-IT int4 + Hadamard β Google Tensor NPU
|
| 2 |
|
| 3 |
-
The exact recipe behind every `.litertlm` in this repo.
|
| 4 |
-
**
|
| 5 |
|
| 6 |
Executable version with guardrails: [`recipe/gemma3_int4hr_recipe.py`](./recipe/gemma3_int4hr_recipe.py).
|
| 7 |
|
|
@@ -70,12 +70,13 @@ Put the compiled prefill_decode into a `.litertlm` together with the embedder, a
|
|
| 70 |
metadata sections, using `litert-lm-builder toml --path model.toml output --path <out>.litertlm`.
|
| 71 |
Check the result with `litert-lm-peek --litertlm_file <out>.litertlm`.
|
| 72 |
|
| 73 |
-
##
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
|
|
|
| 79 |
|
| 80 |
## Per-SoC notes
|
| 81 |
| SoC | Rank-2 | Result |
|
|
|
|
| 1 |
# AOT recipe β Gemma3-1B-IT int4 + Hadamard β Google Tensor NPU
|
| 2 |
|
| 3 |
+
The exact recipe behind every `.litertlm` in this repo. A from-scratch G4 rebuild made this way is
|
| 4 |
+
**bit-identical** to the published G4 build (see "Does your build match?" below).
|
| 5 |
|
| 6 |
Executable version with guardrails: [`recipe/gemma3_int4hr_recipe.py`](./recipe/gemma3_int4hr_recipe.py).
|
| 7 |
|
|
|
|
| 70 |
metadata sections, using `litert-lm-builder toml --path model.toml output --path <out>.litertlm`.
|
| 71 |
Check the result with `litert-lm-peek --litertlm_file <out>.litertlm`.
|
| 72 |
|
| 73 |
+
## Does your build match?
|
| 74 |
+
With the pinned toolchain, the compiled G4 prefill_decode (step 5 output, rank-2) should be
|
| 75 |
+
**560,536,512 B, md5 `f1102df2fdc182d7738bd495ad8f9cf0`**, bit-identical to the one in the published
|
| 76 |
+
G4 bundle. A different md5 means something in your pipeline drifted (most often: quantizing export_hf's
|
| 77 |
+
default int8 copy instead of the float `model.tflite`).
|
| 78 |
+
On device, the rebuild and the published bundle also give word-for-word identical replies to the same
|
| 79 |
+
prompt (greedy decoding).
|
| 80 |
|
| 81 |
## Per-SoC notes
|
| 82 |
| SoC | Rank-2 | Result |
|
README.md
CHANGED
|
@@ -45,21 +45,21 @@ extra_gated_button_content: Acknowledge license
|
|
| 45 |
| SoC | Pixel | File | Status |
|
| 46 |
|---|---|---|---|
|
| 47 |
| **Tensor G4** | Pixel 9 / 9 Pro / 9 Pro XL / 9 Pro Fold | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm` | β
**VERIFIED on device** |
|
| 48 |
-
| **Tensor G3** | Pixel 8 / 8 Pro / 8a | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G3.litertlm` | π‘ **Same program as the verified G4 build;
|
| 49 |
| Tensor G5 | Pixel 10 series | β | π§ in progress (current recipe doesn't compile yet for G5) |
|
| 50 |
| Tensor G6 | Pixel 11 series | β | π§ in progress (current recipe doesn't compile yet for G6; will be verified on device) |
|
| 51 |
|
| 52 |
> [!WARNING]
|
| 53 |
> **Tensor G3 is not yet confirmed on a Pixel 8.** Strong evidence it works: the G3 and G4 compiles
|
| 54 |
> are the **same program**. They differ in only 19 bytes (a device-type tag and its hash), and the G3
|
| 55 |
-
> bundle **runs on a Pixel 9 (G4) and
|
| 56 |
> 8 Pro / 8a, please test it and comment on this repo** (Community tab) with: device, whether it loads
|
| 57 |
> on the NPU, tokens/s, and a sample multi-sentence reply. That's what moves it to β
.
|
| 58 |
|
| 59 |
### Cross-generation DGC compatibility (tested on a Pixel 9 Pro XL / Tensor G4)
|
| 60 |
- **G3 β G4:** the G3 and G4 builds compile to the same program. They differ only in a device-type
|
| 61 |
field (`03` vs `04`) in the embedded compiler options, plus its 8-byte hash. The G3 bundle loads
|
| 62 |
-
and
|
| 63 |
- **G5 β G4:** in an earlier test, a G5-compiled Gemma4-E2B loaded on G4 and its **prefill** executed
|
| 64 |
on the G4 NPU, but **decode did not**. G5 compiles use dual-cluster optimizations that G4 hardware
|
| 65 |
lacks, so newer-generation DGCs are only partly forward-loadable.
|
|
@@ -123,8 +123,9 @@ Full step-by-step in **[`AOT_RECIPE.md`](./AOT_RECIPE.md)**; executable version
|
|
| 123 |
**LMS only, no sharding flag.**
|
| 124 |
6. **Toolchain:** `ai_edge_litert 2.2.0` + the Google Tensor beta compiler (BuildID `87ea807aβ¦`).
|
| 125 |
|
| 126 |
-
|
| 127 |
-
published G4 build
|
|
|
|
| 128 |
|
| 129 |
### Quantization notes
|
| 130 |
|
|
|
|
| 45 |
| SoC | Pixel | File | Status |
|
| 46 |
|---|---|---|---|
|
| 47 |
| **Tensor G4** | Pixel 9 / 9 Pro / 9 Pro XL / 9 Pro Fold | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G4.litertlm` | β
**VERIFIED on device** |
|
| 48 |
+
| **Tensor G3** | Pixel 8 / 8 Pro / 8a | `Gemma3-1B-IT_int4hadamard_aot_ekv4096_G3.litertlm` | π‘ **Same program as the verified G4 build; replies word-for-word identically to the G4 build on G4 hardware β awaiting a Pixel 8 report** |
|
| 49 |
| Tensor G5 | Pixel 10 series | β | π§ in progress (current recipe doesn't compile yet for G5) |
|
| 50 |
| Tensor G6 | Pixel 11 series | β | π§ in progress (current recipe doesn't compile yet for G6; will be verified on device) |
|
| 51 |
|
| 52 |
> [!WARNING]
|
| 53 |
> **Tensor G3 is not yet confirmed on a Pixel 8.** Strong evidence it works: the G3 and G4 compiles
|
| 54 |
> are the **same program**. They differ in only 19 bytes (a device-type tag and its hash), and the G3
|
| 55 |
+
> bundle **runs on a Pixel 9 (G4) and gives word-for-word identical replies to the G4 build**. **If you have a Pixel 8 /
|
| 56 |
> 8 Pro / 8a, please test it and comment on this repo** (Community tab) with: device, whether it loads
|
| 57 |
> on the NPU, tokens/s, and a sample multi-sentence reply. That's what moves it to β
.
|
| 58 |
|
| 59 |
### Cross-generation DGC compatibility (tested on a Pixel 9 Pro XL / Tensor G4)
|
| 60 |
- **G3 β G4:** the G3 and G4 builds compile to the same program. They differ only in a device-type
|
| 61 |
field (`03` vs `04`) in the embedded compiler options, plus its 8-byte hash. The G3 bundle loads
|
| 62 |
+
and gives word-for-word identical replies to the G4 build on G4 hardware, so the G4 runtime does not enforce the tag.
|
| 63 |
- **G5 β G4:** in an earlier test, a G5-compiled Gemma4-E2B loaded on G4 and its **prefill** executed
|
| 64 |
on the G4 NPU, but **decode did not**. G5 compiles use dual-cluster optimizations that G4 hardware
|
| 65 |
lacks, so newer-generation DGCs are only partly forward-loadable.
|
|
|
|
| 123 |
**LMS only, no sharding flag.**
|
| 124 |
6. **Toolchain:** `ai_edge_litert 2.2.0` + the Google Tensor beta compiler (BuildID `87ea807aβ¦`).
|
| 125 |
|
| 126 |
+
Reproducibility check: a from-scratch rebuild with this recipe produces a compiled prefill_decode that is
|
| 127 |
+
**bit-identical** to the published G4 build (md5 `f1102df2β¦`), and gives word-for-word identical replies
|
| 128 |
+
on device.
|
| 129 |
|
| 130 |
### Quantization notes
|
| 131 |
|