---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Thinking-2507
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - qwen3
  - reasoning
  - thinking
pipeline_tag: text-generation
library_name: litert-lm
base_model_relation: quantized
---

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: `com.google.ai.edge.litert:litert`), and `litert-torch`, the renamed `ai-edge-torch`, is its PyTorch converter: a PyTorch model converted unmodified with `litert_torch.convert` matched the original to 4e-7 on a Galaxy S26 ([measured](https://github.com/john-rocky/apple-silicon-llm-bench/blob/main/results/android/tinyhybridnet-three-runtimes.md), LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, qwen3-4b-thinking): Galaxy S26 · LiteRT-LM prebuilt-adac974b · GPU · decode 18.1 tok/s · prefill 263 tok/s · TTFT 3.95 s · all 1707 ops delegated (2026-10-05); Mac Studio M4 Max · LiteRT-LM 0.17.1 · GPU · decode 92.1 tok/s · prefill 1100 tok/s · TTFT 244 ms · all 1707 ops delegated (2026-10-05); Galaxy S26 · LiteRT-LM prebuilt-adac974b · CPU · decode 7.3 tok/s · prefill 56 tok/s · TTFT 18.95 s (2026-10-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen3-4b-thinking/CARD.md

Measured on device (edge-compat, qwen3-4b-thinking-dynamic-wi4b32-afp32): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 11.2 tok/s · prefill 129 tok/s · TTFT 1.65 s · all 1282 ops delegated (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.4 tok/s · prefill 9 tok/s · TTFT 29.40 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/qwen3-4b-thinking-dynamic-wi4b32-afp32/CARD.md

# Qwen3-4B-Thinking-2507 — LiteRT-LM (blockwise int4)

[Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) converted to
the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's
[LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime (the engine behind the official
`litert-community/*` models).

Qwen3-4B-Thinking-2507 is a dense 4B **reasoning model** (`Qwen3ForCausalLM`, 36 layers) that
operates exclusively in thinking mode — it emits a `<think>…</think>` chain before its answer —
so it rides the existing Qwen3 converter and runtime directly.

## Artifact (Block 128)

| | |
|---|---|
| **File** | `model.litertlm` — int4 **block 128** (~2.3 GB) |
| **Quantization** | int4 weights (symmetric) + **OCTAV** optimal-clipping; embeddings INT8 (externalized section) |
| **Compute** | integer |
| **Context (KV cache)** | 4096 |
| **Base model** | Qwen/Qwen3-4B-Thinking-2507 |

**2026-10-07: chat template made parts-safe.** `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` was replaced by a copy whose embedded Jinja template reads message content both as a string (what litert-lm 0.17.1 passes) and as a list of typed parts (what 0.18.0 passes). On litert-lm 0.18.0, the previous file stopped at the template with `tried to use + operator on unsupported types string and sequence`; on 0.17.1 it answered. Only the template section changed: the TFLite graph and weights, the tokenizer and the executor metadata are byte-identical, and for string content the new template renders the same prompt as the old one, byte for byte. The new file is still 2,274,193,168 bytes, sha256 `110098cf…` (full value in `litertlm_manifest.json`); the previous file (Hub revision `6c4863d9`) had sha256 `a412d77d…`. Tested on macOS: with litert-lm 0.18.0 the new file answers on CPU and scores 8/8 sanity-gate questions on the Mac GPU; with 0.17.1 it gives the same text as the previous file. Tested on a Galaxy S26 with `litert_lm_main` built from the v0.18.0 tag: the new file answers on the GPU (OpenCL) and on the CPU.

**2026-10-06: GPU graph rewrite.** `model.litertlm` was replaced by a file with a rewritten prefill/decode graph. In each layer, the two DYNAMIC_UPDATE_SLICE KV-cache writes became one `odml.cache_update` composite, and the BATCH_MATMUL attention products became `odml.runtime_bmm` composites. This is the graph shape litert-torch exports with `--apply_gpu_composites`; the previous file was exported without that flag. The graph also gains a signature input, `param_tensor` (INT32[1,1,1,7]), whose start/end values the runtime fills. The attention mask stays a FLOAT32 input. A 16-token prefill signature, `prefill_16`, was added as a copy of `prefill_128` that shares its weight buffers. Every other bundle section, and the weight region of the graph, is byte-identical to the previous file (checked per section and per buffer). The new file is 2,477,192,112 bytes, sha256 `8a8a6a419a5570b3f4814d358fb795c0dcb6931206c7dae29480f47564c3f04d`. The previous file (Hub revision `92751be3c031`) was 2,474,357,680 bytes, sha256 `356b2d7778d27d55fbb0d2e0c5e8801e9de136db59849c41c882bd97333840d5`. On CPU, `prefill_16` gives output bit-identical to `prefill_128` on the same tokens. With and without `prefill_16`, the answers to 9 gate prompts (8 sanity-gate questions plus one long prompt) are byte-identical, on CPU and on the Mac GPU with fp32 activations. On the Mac GPU with fp32 activations, the rewritten graph without `prefill_16` also matches the previous file byte for byte on all 8 questions. On the Mac GPU with the default fp16 activations, 5 of 30 GSM8K answer texts stay byte-identical. The scores hold: both files answer 8/8 sanity-gate questions and 26/30 GSM8K questions (max tokens 2048, greedy), and each solves one GSM8K question the other misses. The Raspberry Pi 5 rows and the iPhone 17 Pro note further down were measured on the previous file. The commands are under Conversion.

Mac Studio M4 Max, litert-lm 0.17.1 `benchmark --cache no`, both files measured in the same window (median of 3 processes on GPU, 2 on CPU; GPU is WebGPU on Metal with fp16 activations, CPU is XNNPACK):

| Measure | Previous file | New file | New / previous |
|---|---:|---:|---:|
| GPU decode (256-token prompt, 256 decode tokens) | 66.6 tok/s | 92.1 tok/s | ×1.383 |
| GPU prefill (256-token prompt) | 984 tok/s | 1100 tok/s | ×1.118 |
| GPU TTFT (16-token prompt) | 0.152 s | 0.037 s | ×0.247 |
| CPU decode (256-token prompt, 256 decode tokens) | 18.5 tok/s | 18.4 tok/s | ×0.997 |
| CPU TTFT (16-token prompt) | 1.444 s | 0.478 s | ×0.331 |

## Artifact (Block 32)
| File | Quantization Recipe | Context | Size |
|---|---|---:|---:|
| `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` | dynamic_wi4b32_afp32 (block-32) | 4096 | 2.1 GB |

## Conversion Notes

`Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` is a dynamic INT4 variant (block-32 weights, FP32 activations). It was converted through the LiteRT Torch (`litert-torch`) path and quantized with AI Edge Quantizer. This artifact incorporates LiteRT-LM GPU graph optimizations, including composite ops for RoPE, fused QKV, and fused Gate/Up projections, and is configured with static prefill memory allocation. 

> **Update (2026-08-31) — thought channel and think pre-fill repaired in place.** Both bundles received two metadata-only repairs today (weights byte-identical each time, verified section by section). First, the `thought` channel (`<think>\n` / `\n</think>`) was declared in the bundle metadata — without it the runtime streams raw reasoning inline into the answer and silently ignores any thinking budget. Second, the generation prompt now **pre-opens the think block exactly as the upstream Qwen3-4B-Thinking-2507 template does** (`<|im_start|>assistant\n<think>\n`): `model.litertlm`'s structured assistant prefix and the block-32 file's embedded template both previously ended at `assistant\n`, leaving the model to emit `<think>` on its own — a discipline quantized reasoning models lose first. After the repair `model.litertlm` scores 8/8 on the 8-question sanity gate on both macOS backends (CPU and Metal GPU, max-tokens 2048), and the block-32 file scores 8/8 on CPU.

## ⚠️ It's a reasoning model — give it room to think

This model generates a `<think>…</think>` reasoning chain, then the answer. **Run it with
`max_tokens` ≥ 2048** — at a short limit it gets cut off mid-thought and never reaches the answer.
(All quality numbers below were measured at 2048.)

**Thinking cannot be switched off.** On litert-lm 0.18.0 `ThinkingConfig(enable_thinking=False)` has no effect on both files: with a 400-token cap the thought channel uses the whole cap and no answer text appears (measured 2026-10-13). Set `thinking_token_budget` (300 answered 17 × 23 under a 700-token cap) and a cap above it.

## Performance

All rows are for the current `model.litertlm` (the 2026-10-06 file).

**Mac Studio M4 Max.** litert-lm 0.17.1 (pip) CLI, `litert-lm benchmark --cache no`, 256 prompt / 256 decode tokens. Each value is the median of separate processes: 3 on GPU, 2 on CPU. The 16-token TTFT comes from separate runs with a 16-token prompt and 32 decode tokens.

| Backend | Prefill (256 tokens) | Decode (256 tokens) | TTFT (256-token prompt) | TTFT (16-token prompt) | Init |
|---|---:|---:|---:|---:|---:|
| GPU (WebGPU on Metal, fp16 activations) | 1100 tok/s | 92.1 tok/s | 0.24 s | 0.037 s | 2.2 s |
| CPU (XNNPACK) | 113 tok/s | 18.4 tok/s | 2.57 s | 0.478 s | 2.1 s |

Init is the `Init time` the CLI prints; that value includes both `Init Total` and the phases inside it, so it counts engine creation twice ([LiteRT-LM #4057](https://github.com/google-ai-edge/LiteRT-LM/issues/4057)).

**Samsung Galaxy S26** (SM-S942Q, Android 16). `litert_lm_advanced_main` from a LiteRT-LM main build of 2026-09-18, 1024 prompt / 256 decode tokens. The GPU runs use `--disable_cache=true` (cold init); the CPU runs use the XNNPACK weight cache, which init writes. Each run is one process: 3 iterations on GPU, 2 on CPU, and 3 in each separate 16-token run (16 prompt / 32 decode tokens). Each cell is the range over those iterations. In the 1024-token GPU run the first iteration is the fastest and the third the slowest (cold to warm). In the 16-token runs the first iteration has the longest TTFT. Init and peak memory (the process high-water mark, `VmHWM`) are from the 1024-token run.

| Backend | Prefill (1024 tokens) | Decode (256 tokens) | TTFT (1024-token prompt) | TTFT (16-token prompt) | Init | Peak |
|---|---:|---:|---:|---:|---:|---:|
| GPU (OpenCL, fp16 activations) | 130–264 tok/s | 13.9–20.2 tok/s | 3.93–7.95 s | 0.14–0.15 s | 7.9 s | 1.17 GB |
| CPU (XNNPACK, 4 threads) | 47–65 tok/s | 7.2–7.4 tok/s | 15.78–22.13 s | 0.40–1.29 s | 2.8 s | 4.08 GB |

### Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, **max_tokens 2048**, identical prompt
and answer-extraction for every row).

| Configuration | GSM8K |
|---|---|
| bf16 (reference) | 90.0% |
| **LiteRT int4 — block 128** | **86.0%** (−4 pt) |

int4 is at parity (−4 pt). Note: evaluating a reasoning model at a short token budget badly
understates int4 — the longer int4 reasoning chains get truncated before the answer; benchmark
reasoning models with `max_tokens` ≥ 2048.

**Why block 128 (and not block 32)?** For this reasoning model the block-32 build degraded more
(−9 pt) and produced corrupted output under the iPhone GPU delegate, while **block 128 is robust
on every backend, ~40 % faster to decode** (¼ the dequant scales — which matters when generating
long `<think>` chains), and stays at −4 pt parity. So block 128 is the build we recommend; the block-32 file is also published (see **Artifact (Block 32)** above), and both bundles run fully delegated on the Galaxy S26 GPU.

The block-32 file's GPU corruption is not iPhone-specific: on **macOS Metal** it degenerates too (question-echo loops, truncated reasoning; measured 2026-08-31 on the file both before and after the metadata repair, so it is pre-existing and unrelated to the repair). **Use the block-32 file on CPU**; on Android GPU it delegated and generated in the Galaxy S26 gate above.

## Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

**`model.litertlm` (the 2026-10-06 file).** On a **Samsung Galaxy S26** (SM-S942Q, Android 16) with `litert_lm_advanced_main` from a LiteRT-LM main build of 2026-09-18, the three transformer signatures run fully on the OpenCL GPU delegate (`LITERT_CL`), one partition each:

| signature | nodes on `LITERT_CL` |
|---|---:|
| `prefill_128` | 1707 / 1707 |
| `prefill_16` | 1707 / 1707 |
| `decode` | 1631 / 1631 |

XNNPACK takes 1 of the 4 nodes in `decode_embedder` and 1 of the 4 nodes in `prefill_embedder_128`. On the GPU the file answered 5 of 5 gate prompts correctly. Speed and peak memory are in the Galaxy S26 table under Performance.

**Block-32 file.**

| file | GPU backend | delegation | peak |
|---|---|---|---:|
| `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` | runs | 3764 / 3764 ops across 3 subgraphs on `LiteRT GPU` | 1627 MB |

Measured on a **Samsung Galaxy S26** (SM-S942Q / SM8850, Android 16) with `litert_lm_advanced_main` from litert-lm 0.16.0, `--backend=gpu --sampler_backend=cpu`, prompt `What is the capital of France?`. Peak is the process high-water mark (`VmHWM`) sampled during that same run. Gated 2026-08-24.

The op counts above are the `LiteRT GPU` partitions.

GPU wiring, including the Gallery import toggle: [GPU guide](https://github.com/john-rocky/hf-to-litertlm/blob/main/docs/android-gpu.md).

## Usage

```bash
# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A bat and a ball cost \$1.10. The bat costs \$1.00 more than the ball. How much is the ball?"
```

The `.litertlm` bundle carries the tokenizer and prompt template (Qwen3 ChatML —
`<|im_start|>role\n…<|im_end|>`, stop token `<|im_end|>`), so no separate tokenizer files are
needed. The model produces a `<think>…</think>` block followed by its answer.

## Run on Android

> **Update (July 2026):** [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) **v1.0.16+** can import litert-lm models **directly from Hugging Face** inside the app (tap **+**) — no computer or `adb` needed. The manual steps below are only required on older builds or for sideloading a local file.

The official **[Google AI Edge Gallery](https://github.com/google-ai-edge/gallery)** app runs
`.litertlm` models on-device:

1. Install a **recent** Gallery (package `com.google.ai.edge.gallery`, 1.0.15+ supports `.litertlm`).
2. Download `model.litertlm` and push it: `adb push model.litertlm /sdcard/Download/`
3. In the app tap **+**, pick the file, choose the **GPU** backend, and raise the max-tokens setting (≥2048).
4. Chat — the bundle already carries the tokenizer and Qwen3 chat template.

A 4B int4 build needs ~2.5 GB free RAM; reboot the phone first if memory is tight.

In a Flutter app, [flutter_edge_ai_litertlm](https://pub.dev/packages/flutter_edge_ai_litertlm) (1.14.0) installs this repo's bundle through `FlutterEdgeAi.resolveHuggingFace('litert-community/Qwen3-4B-Thinking-2507')`, which reads the repo's `litertlm_manifest.json` and picks the file and backend for the device; in React Native, [react-native-litert-lm](https://www.npmjs.com/package/react-native-litert-lm) (0.8.0) does the same through `resolveFromManifest()`. Neither route is device-verified by this card; each package's README states its tested platforms.

## Run on desktop (LiteRT-LM CLI)

The same `.litertlm` bundle runs on macOS / Linux / Windows with the official
[LiteRT-LM CLI](https://github.com/google-ai-edge/LiteRT-LM) — including as a
local **OpenAI-compatible API server**:

```bash
pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Qwen3-4B-Thinking-2507 model.litertlm qwen3-4b-thinking-2507
litert-lm run qwen3-4b-thinking-2507     # interactive chat in the terminal
litert-lm serve
```

## Run on iPhone

Verified on **iPhone 17 Pro** (LiteRT-LM Swift runtime): loads and generates at ~14 tok/s (previous file).

## Conversion

Converted with the **official** [`litert-torch`](https://github.com/google-ai-edge/litert-torch)
converter — a standard `Qwen3ForCausalLM`, so it uses the existing Qwen3 path with no custom graph
code. Recipe: **blockwise-128 int4 + OCTAV** (INT4 weights, block 128, symmetric, OCTAV
optimal-clipping), embeddings INT8, KV cache 4096.

```python
from litert_torch.generative.export_hf.export import export
export(
    model="Qwen/Qwen3-4B-Thinking-2507",
    output_dir="out",
    quantization_recipe="qwen3_int4_block128_octav.json",  # blockwise-128 int4 + OCTAV, int8 embeddings
    cache_length=4096,
    externalize_embedder=True,
)
```

**Graph rewrite (2026-10-06).** The current `model.litertlm` was built from the previous file (Hub revision `92751be3c031`) with two tools in `tools/gpu_graph/` of [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm):

```bash
python tools/gpu_graph/gpu_graph_retrofit.py previous.litertlm retrofit.litertlm --decomp exporter --mask add_bcast
python tools/gpu_graph/prefill_bucket_clone.py build retrofit.litertlm new.litertlm --lengths 16 --source prefill_128 --report bucket.json
```

`previous.litertlm` is the previous file, `new.litertlm` the result published here as `model.litertlm`; `LITERT_LM_CLI` names the `litert-lm` CLI the scripts use to unpack and pack the bundle.

`gpu_graph_retrofit.py` changes only the prefill/decode TFLite graph, and every original weight buffer keeps its bytes. With `--mask add_bcast`, the attention mask is applied as an ADD on a [bk, g, T, C] view of the logits, with the FLOAT32 mask broadcast. `prefill_bucket_clone.py` copies an existing prefill signature (`prefill_128`) at a new length (16); the copy shares every weight buffer, so the file grows by graph structure only. A rebuild with these tools has every bundle section byte-identical to the shipped file. Only the bundle header (uuid and timestamp written by `litert-lm pack`) differs, so the whole-file sha256 differs.

## Raspberry Pi 5 (CPU) (previous file)

The `model.litertlm` row was measured on the previous file (sha256 `356b2d7778d27d55fbb0d2e0c5e8801e9de136db59849c41c882bd97333840d5`) and not re-measured after the 2026-10-06 graph rewrite; the weights are byte-identical in the new file, and the new file's Mac CPU speed is ×0.997 (decode) and ×1.002 (prefill) of the previous file's, so the row is expected to hold.

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with [`litert-lm benchmark`](https://github.com/google-ai-edge/LiteRT-LM) 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, `--cache memory` (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (`vcgencmd get_throttled` stayed `0x0`). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|---:|---:|---:|---:|
| `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` | 8.9 (8.8–9.0) | 1.4 (1.4–1.4) | 29.4 s | 4.4 GB |
| `model.litertlm` | 10.6 (10.5–10.7) | 1.5 (1.5–1.5) | 25.5 s | 3.9 GB |

## Known runtime issues

**External weights (since 2026-10-30).** In `model.litertlm` and `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm`, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file ([LiteRT-LM#3986](https://github.com/google-ai-edge/LiteRT-LM/issues/3986)). The earlier files with the weights inside the graph stay at revision [`9df72e0`](https://huggingface.co/litert-community/Qwen3-4B-Thinking-2507/tree/9df72e073feaaeaa00911d9288ddd39f74608272).

The bundle's chat template opens a thought in every answer, so after each answer the runtime rebuilds the cache from the re-rendered history at the next turn: the second turn is free of the defect below when the conversation opens with a user message, and the same source shows a zero embedding at another token from the third turn on (not measured). The defect: after an answer without a thought (no `<think>` … `</think>` block), from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because `model.litertlm` holds the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` has no embedder section (the decoder looks the tokens up itself) and is not affected. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders the `Qwen3_4b_thinking_dynamic_wi4b32_afp32.litertlm` file's chat template with an empty message list, and the template reads `messages[0]` without checking that the list is non-empty, so the render fails with `Failed to apply template: undefined value`. This happens in two places: when the preface is prefilled at creation (`prefillPrefaceOnInit = true` in Kotlin, `prefillPrefaceOnInit: true` on the web, `SetPrefillPrefaceOnInit(true)` in C++; the default is false), where creating the conversation fails with that error, and in the preface render (`renderPrefaceIntoString()` in Kotlin and Swift, `RenderPrefaceIntoString()` in C++, `litert_lm_conversation_render_preface_to_string` in the C API). The same happens with extra context (such as `enable_thinking`) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it. `model.litertlm` carries no Jinja chat template and is not affected: the runtime's qwen3 template only loops over the messages.

## License

Apache-2.0, inherited from the base model
[Qwen/Qwen3-4B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507).