granite-3.0-1b-a400m-instruct — LiteRT-LM

ibm-granite/granite-3.0-1b-a400m-instruct converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime: one int8 file of IBM's 32-expert mixture-of-experts model, computing every expert for every token. Converted with litert-torch at commit 0d3869c and tested on litert-lm 0.18.0 on a Mac Studio M4 Max, where it decodes at 146.2–146.5 tok/s on the GPU and 53.0–53.1 tok/s on the CPU (measured 2026-10-21).

IBM describes Granite-3.0-1B-A400M-Instruct as a 1B-parameter model finetuned from Granite-3.0-1B-A400M-Base on open-source instruction datasets with permissive licenses and internally collected synthetic data, with supervised finetuning, reinforcement-learning alignment and model merging; released October 21, 2024. It is a decoder-only sparse mixture-of-experts transformer: 24 layers, hidden size 1024, 16 attention heads with 8 KV heads, 32 experts per layer of which 8 are chosen for each token (SwiGLU, expert hidden size 512), a 49,155-token vocabulary with tied embeddings and a 4096-token context (2048 in this file); 1,334,628,352 bf16 parameters by the Hub's count, 1.3B with 400M active per token in IBM's table. IBM lists twelve languages, from English to Chinese.

File	Recipe	Size	sha256
granite-3.0-1b-a400m-instruct_int8.litertlm	int8 dynamic on linears, float activations; each layer's 32 experts as two fully connected layers; fp32 GPU activations declared	1.37 GB (1,372,745,360 bytes)	b0b786a5672c25d4aeb8ed90b7363294703809a1f95a5dfb8fb3dcf5db11d39f

litert-torch's sparse litert_moe export is out for two reasons: it computes its experts with GELU-tanh while this model's experts use SiLU, and LiteRT-LM 0.18.0's GPU refuses the moe op it writes (on a two-layer test model: moe only supports activation='gelu'.). The file therefore takes litert-torch's litert_moe_sequential route, which computes all 32 experts of a layer for every token with the model's SiLU (the routing weight is zero for the 24 experts outside the token's top 8): a token costs the compute of all 1.33B parameters, not 400M. Each layer's 32 expert FCs are fused into two FCs (Conversion notes).

On GSM8K (n=100) bf16 scored 34 and this file 35 in each of three passes on the Mac GPU (table in Accuracy). Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
GPU	2.06 and 2.07 (two runs)	12.25, 12.35 and 12.34 (three runs, each with the GPU clock lowered by heat)	28.24 and 28.40 (two runs)
CPU	6.48 and 6.49 (two runs)	14.12 and 13.57 (two runs)	26.59 and 29.15 (two runs)

On the Galaxy S26 the GPU took less time than the CPU in every run; on the Pixel 8a the CPU took 1.6 % less on average, though its second run was slower than both GPU runs (Performance).

Correctness

8-question check on the Mac M4 Max (the questions with " Answer briefly." appended, --top-k 1, one litert-lm run process each): GPU 8/8, CPU 8/8 (one CPU thread), bf16 (transformers) 8/8. The first eight tokens of each answer equal the bf16 model's greedy answer in 8 of 8 on the GPU and 7 of 8 on the CPU (the rhyme, where the CPU's answer and bf16's share the first four tokens).

On the Mac GPU every node of the three signatures runs on WebGPU (1,365 of 1,365 in each prefill signature, 1,266 of 1,266 in decode, one partition each; adapter Apple M4 Max, backend Metal), and the GPU run prints no TopK warning (Conversion notes).

On the Galaxy S26 and the Pixel 8a (Performance), the same eight prompts through litert_lm_advanced_main with its default sampler (the bundle sets none) scored 8/8 on the GPU and 7/8 on the CPU of each phone; the CPUs' rhyme answer, "Sweet, wilted violets are.", has no "blue". The first eight tokens equal bf16's in 8 of 8 answers on each GPU and 7 of 8 on each CPU.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction that asks for a final #### <number> line, the bundle's template (no system turn), greedy (top-k 1), at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API, Mac M4 Max); bf16 in transformers on MPS with the same prompts. The model wrote the #### line in none of the replies (0 of 100 in every configuration, bf16 included), so a reply is scored by the last number that follows "answer", "total" or "result" with "is", ":" or "=" (a \boxed{} number would come first; none appeared), else by the last number in the reply. Scores are comparable only within this table; the model-index in IBM's README (an entry named granite-3.0-2b-instruct) lists GSM8K 26.31 for a setup it does not state.

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	34
this file, Mac GPU (fp32 activations, declared in the file)	35 / 35 / 35
the same weights, Mac GPU with the runtime's default fp16 activations	28 / 29 / 29 (third pass: one question re-run, below)
the same weights, Mac CPU	39 / 35

The three fp32 GPU passes give the same verdict on every question (97 to 99 of the 100 replies byte-identical between passes); between an fp16 pass and an fp32 pass 23 to 25 questions change sides. In the third fp16 pass one question's process logged a WebGPU validation error; re-run in a fresh process it was answered correctly, which turned that pass's 28 into 29. The file therefore declares fp32 GPU activations (prefer_activation_type fp32): measured back to back against the same weights without the declaration, the Mac GPU then decodes 5.4 % slower and prefills 17.7 % slower. The two CPU passes used the same weights before the declaration was added, and the declaration does not reach the CPU path (on one CPU thread the declared file gives the same eight check answers byte for byte); the two passes differ by 4 points with 18 questions changing sides, so differences of a few points on these 100 questions are within the run-to-run spread.

Usage

litert-lm run ./granite-3.0-1b-a400m-instruct_int8.litertlm --prompt "Which is larger: 0.9 or 0.11? Answer briefly."
litert-lm run ./granite-3.0-1b-a400m-instruct_int8.litertlm --prompt "Which is larger: 0.9 or 0.11? Answer briefly." --backend gpu --cache no
litert-lm benchmark ./granite-3.0-1b-a400m-instruct_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 1 --cache no --backend gpu
litert-lm benchmark ./granite-3.0-1b-a400m-instruct_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 1 --cache no --backend cpu

Both run commands answered "0.9 is larger than 0.11." with no template token (such as <|end_of_text|>) in the output; with --top-k 1 added, each backend gave the same answer, and in the 8-question check the Galaxy S26 and the Pixel 8a gave the same answer on both backends. The bundle sets no sampler, and run passes one only when you give --top-k, --top-p, --temperature or --seed, so these answers come from the engine's default sampler. Without --backend the first command ran on the CPU, and without --cache it wrote an XNNPACK weight cache (the CPU backend's packed weights) next to the model (1,346,661,944 bytes). When we ran the GPU benchmark command for this card it printed decode 79.71 tok/s, about half the Performance table's rate, with the usual prefill (3216.68 tok/s) and no error message. We found no cause. Three later runs of the same command printed decode 143.66, 143.80 and 144.05 tok/s, within 2 % of the table. The CPU command printed decode 51.92 tok/s, 2 % under the table.

The chat template is IBM's (the chat_template of the pinned commit's tokenizer_config.json) with three additions for litert-lm 0.18: a message's content may be a string or a list of text and tool-response parts; the tools block is written by a macro that gives the bytes of tojson(indent=4) (the 0.18 runtime's tojson takes no arguments); and a tool role renders as IBM's tool_response turn. On litert-lm 0.18.0 the template renders the same text as IBM's template does in transformers for 15 of 15 conversations without tools and 20 of 20 with tools (two further test conversations with a float in a tool schema differed only in how the float is written: 1e-05 in transformers, 0.00001 in the runtime). No start token (bos is eos for this tokenizer and add_bos_token is false); the stop token is <|end_of_text|> (id 0), plus 14 strings the exporter derives from the template's turn ending; no thinking declared; the bundle sets no sampler.

Tool calling: the bundle's model type (generic_model) has no tool-call parser, so a call arrives as reply text, <|tool_call|> followed by a JSON list, for example <|tool_call|>[{"name": "calculator", "arguments": {"expression": "48213 / 37"}}] (this file, Mac GPU); the app parses it. In our tests the model started a call only when the history already contained one earlier call and its result (a round trip): in transformers (bf16, greedy) 0 of 24 single-tool prompts began with <|tool_call|>, 0 of 24 again with Granite 3.1's tool system prompt, and 12 of 12 with one earlier round trip; on those prompts this file's GPU began with <|tool_call|> (3 of 3 checked). When the tool's result was sent back (same weights, before the fp32 declaration), the model's next reply used it whenever the call had parsed as JSON (CPU 12 of 12; GPU with fp16 activations 11 of 11); the CPU answered "It is 41 degrees Celsius and there is violet hail in London today." for a tool that returned 41 °C and violet hail, and the GPU wrote a call whose JSON did not parse in 1 of 12 processes.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the two benchmark commands above (1024-token prefill, 256-token decode, context 2048), one warmup and one measured run per process, two processes per backend, each backend's pair run back to back with nothing else measuring on the Mac; GPU is WebGPU on Metal. Init is the Init time the CLI prints, which counts engine creation twice (LiteRT-LM #4057).

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
GPU, run 1 / run 2	3245.07 / 3228.70	146.53 / 146.15	0.322 / 0.324	1.33 / 1.29
CPU, run 1 / run 2	617.89 / 617.85	53.11 / 52.99	1.676 / 1.676	4.32 / 4.31

Peak memory footprint (macOS time -l): 2.55 GB on the GPU, 4.70 GB on the CPU (maximum resident set 1.71–1.75 GB and 6.29–6.30 GB).

Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, Android 16, MemTotal 11,389,756 kB) and Pixel 8a (Google Tensor G3, Mali-G715, Android 16, MemTotal 7,754,700 kB), 2026-10-21, the same workload:

• Binary: litert_lm_advanced_main built from source at the LiteRT-LM v0.18.0 tag (sha256 cf363607…, 40,202,976 bytes). It is given no activation type (it prints activation_data_type: Not set), so the engine takes the file's fp32 declaration.
• Command: --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3, one process per run. Each rate is the mean of iterations 2 and 3 of that process. A phone's CPU and GPU were measured in separate runs of the harness.
• Backends: GPU = LiteRT's OpenCL backend, which took every node of the three signatures (1,365, 1,365 and 1,266 nodes, one partition each); CPU = XNNPACK at the binary's default thread count.
• Before each run: 123 to 254 s of rest, no CPU or GPU clock ceiling lowered, and Android's thermal status at 0 or 1.
• Init is the Init Total the binary prints, with the caches an earlier process wrote next to the model (CPU: XNNPACK's weight cache, 1,346,661,944 bytes on the Galaxy S26 and 1,345,804,216 on the Pixel 8a; GPU: the compiled programs, plus a 1,342,110,512-byte weight cache on the Pixel 8a). Peak is the process's VmHWM.

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)	Peak (GB)
GPU, run 1 / run 2 / run 3	351.305 / 352.895 / 346.705	27.430 / 27.105 / 27.285	2.955 / 2.940 / 2.990	1.52 / 1.73 / 1.64	0.54 / 0.54 / 0.54
CPU, run 1 / run 2	355.190 / 368.095	22.785 / 23.725	2.935 / 2.835	0.09 / 0.08	2.10 / 2.10

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)	Peak (GB)
GPU, run 1 / run 2	128.405 / 128.295	12.630 / 12.535	8.055 / 8.065	4.73 / 3.48	1.36 / 1.36
CPU, run 1 / run 2	132.295 / 129.405	13.580 / 12.055	7.855 / 8.035	0.38 / 0.30	2.07 / 2.07

Peak in kB (GB = kB × 1,024 / 10^9): Galaxy S26 GPU 525,092 / 526,484 / 526,756, CPU 2,053,544 / 2,053,984; Pixel 8a GPU 1,331,700 / 1,331,380, CPU 2,023,768 / 2,022,632.

On the Galaxy S26 the GPU's clock ceiling fell from 1,300 MHz during every run (first seen 35, 20 and 25 s into the 39–40 s runs; 1,200, 1,000 and 1,050 MHz at the end), so its rates include throttled iterations; we did not get an unthrottled 256-token run. With 64 decoded tokens, in two runs whose GPU clock held, the file decoded at 26.805 and 26.795 tok/s and prefilled at 331.270 and 343.420 tok/s. The harness read the clocks every 2–3 s during a run. The CPU runs had a lowered CPU clock ceiling in 17 of 18 and 16 of 17 samples on the Galaxy S26 and in 9 of 25 and 12 of 27 on the Pixel 8a. On the Pixel 8a the CPU's decode fell within each process (from 15.41 to 12.85 and from 15.17 to 10.48 tok/s between iterations 1 and 3), while its two GPU runs stayed within 1 % of each other. With the caches removed, the first process's init took 1.78 s (Galaxy S26) and 3.44 s (Pixel 8a) on the CPU, 2.79 to 4.86 s on the Galaxy S26 GPU (three first starts) and 24.80 and 30.01 s on the Pixel 8a GPU, where the runtime logs that weight preparation on the GPU is disabled for Mali GPUs.

Cost of the fp32 declaration on the phones' GPUs, against the same weights with the runtime's default fp16 activations (one session per phone, runs in the order fp16, fp32, fp32, fp16, each started unthrottled): the Galaxy S26 (64 decoded tokens) prefills at 0.330× and decodes at 0.841× the fp16 rate, the Pixel 8a at 0.496× and 0.873×; peak process memory +0.10 GB and +0.45 GB; GPU memory (whole device) +0.44 GB and +0.45 GB. The Pixel 8a's GPU weight cache is 1,342,110,512 bytes with the declaration and 1,338,374,960 without it, so the fp32 activations add no fp32 copy of the weights.

Not measured on iOS. No NPU row: the file is a dynamic-range int8 graph and was not tried on an NPU.

Conversion notes

• Source and stack: ibm-granite/granite-3.0-1b-a400m-instruct at revision ffec3c35 (every file checked against the Hub's record); litert-torch at git commit 0d3869c, transformers 5.20.1, torch 2.14.0, ai-edge-quantizer-nightly 0.10.0.dev20261019, litert-converter 0.6.0.dev20261016, ai-edge-litert-nightly 2.4.0.dev20261018; litert-lm 0.18.0 for the activation-type step. granitemoe_work/convert_granite30_1b_a400m.sh in hf-to-litertlm is the one command; run as written on 2026-10-21 it rebuilt all four sections of this file byte for byte (only the packer's uuid and creation timestamp in the header differ).
• Experts as two fully connected layers: litert-torch's litert_moe_sequential route runs each expert as a separate chain of ops (10,889 nodes in the decode signature of that export). The recipe replaces it with the same sum written as two FCs per layer, gate_up of all 32 experts as one [32768, 1024] weight and down of all 32 as one [1024, 16384] weight, with each expert's routing weight multiplied into its slice before the down projection (1,266 nodes in this file's decode signature). On a two-layer granitemoe model with random weights (hidden 128, 8 experts, top 2), the fused graph with the SiLU experts matched PyTorch's fp32 logits at all 20 positions checked (the same argmax everywhere, largest difference 5.1e-7 of the largest logit); against a PyTorch reference whose experts use GELU-tanh instead, the same graph is 5.2e-3 off, so the check tells the two activations apart. In one measurement window the fused graph decoded at 153.1 tok/s on the Mac GPU against 63.6–63.8 for the per-expert graph, and at 42.7 and 53.6 tok/s on the CPU against 30.5–30.6.
• Decode logits padded to 49,156 columns: one column of the constant -1e4 is appended to the decode signature's logits (the vocabulary is 49,155). Without it, every GPU run printed "TopK requires src tensor C dimension to be divisible by 4." and, where the CPU began a reply with <|tool_call|> (id 49154), the GPU began with other text. With it the real 49,155 columns are bit-identical to the unpadded export's on the CPU (8 prompts, 239 positions) and the GPU begins those replies with <|tool_call|>.
• Quantization: int8 = dynamic_wi8_afp32 (channelwise int8 weights, float activations); in the fused down projection one int8 scale per output channel covers the inputs of all 32 experts, coarser than per-expert scales; the 8-question check still passes on both Mac backends. Blockwise int4 (OCTAV, block 32 and block 64) on the per-expert graph was tried and not shipped: its answers drifted from bf16's (first eight tokens equal in 3 and 4 of 8 answers against 7 of 8 for int8 on the same graph).
• Order: export (prefill signatures 128 and 1024 plus decode over a 2048-position KV cache; the checkpoint's tokenizer.json packed byte for byte; the template above; PYTHONHASHSEED=0, because the exporter collects the stop tokens in a Python set and writes them in its iteration order) → scripts/set_activation_type.py --type fp32 (the prefer_activation_type item on the TFLite section; every section's bytes unchanged).

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo carries the license in its README metadata and no separate LICENSE file). Changes from the original work: exported to the LiteRT-LM format and quantized to int8 as described above, each layer's experts fused into two fully connected layers (the same computation), the decode logits padded by one constant column, the chat template extended for litert-lm 0.18, fp32 GPU activations declared. This conversion was made and is published independently of IBM.