Gemma-3-1B on the Tensor G4 NPU — the rank-2 FC surgery
The compiler-bug workaround that makes Gemma-3/4 AoT-compile to the Tensor G4 edgetpu, plus working-model notes.
The rank-2 FULLY_CONNECTED surgery — the key enabler (affects G4 AND G5)
The DarwiNN AoT compiler OP_FAILEDs (file.cc:4780) on rank-3 FULLY_CONNECTED ops.
Gemma-3 (and Gemma-4) emit these ~360× across the model, so the AoT compile fails outright
without a fix.
Fix: wrap every rank-3 FC as RESHAPE → (rank-2) FC → RESHAPE — same math, rank the
compiler accepts. Script: recipe/fc_rank2_surgery.py (surgery(in_tflite, out_tflite)), run
after quantization, before aot_compile. ~360 FCs rewritten for gemma-3-1b. The G5 uses the same
compiler family, so the surgery carries over to G5 as well.
Profile
AoT (int4 per-channel + Hadamard): 100% NPU, prefill-monster (660 tok/s prefill),
power-efficient — best for long-input / RAG. Decode is dispatch-bound rather than
compute-bound, so it sits around 16 tok/s regardless of prefill length.
Behavior note: tone/"feel" differences vs a reference are runtime config (system prompt + sampling), not the quant — a faithful int4 copy matches the reference once system prompt and top-k / top-p / temperature match.
On quantization support tables
Don't trust a fixed "what quantization compiles" matrix, including any you find in older copies of this repo. Per-quant support is compiler-build-dependent: a result from one toolchain build does not generalize to another, and rows that read "unsupported" have repeatedly turned out to be one build's behaviour rather than a hardware or format limit.
Retest against your own target compiler rather than trusting a table.
Runtime requirement (brief)
int4 AoT DGCs require a compatible GoogleTensor dispatch library to load on-device; a stock app's dispatch may reject them as a firmware mismatch.