# Gemma-3-1B on the Tensor G4 NPU — the rank-2 FC surgery

The compiler-bug workaround that makes Gemma-3/4 AoT-compile to the Tensor G4 edgetpu, plus
working-model notes.

---

## The rank-2 FULLY_CONNECTED surgery — the key enabler (affects G4 AND G5)

The DarwiNN AoT compiler **`OP_FAILED`s (`file.cc:4780`) on rank-3 `FULLY_CONNECTED` ops.**
Gemma-3 (and Gemma-4) emit these ~360× across the model, so the AoT compile fails outright
without a fix.

**Fix:** wrap every rank-3 FC as `RESHAPE → (rank-2) FC → RESHAPE` — same math, rank the
compiler accepts. Script: `recipe/fc_rank2_surgery.py` (`surgery(in_tflite, out_tflite)`), run
after quantization, before `aot_compile`. ~360 FCs rewritten for gemma-3-1b. The G5 uses the same
compiler family, so the surgery carries over to G5 as well.

---

## Profile

**AoT (int4 per-channel + Hadamard):** ~100% NPU, prefill-monster (~660 tok/s prefill),
power-efficient — best for long-input / RAG. Decode is dispatch-bound rather than
compute-bound, so it sits around 16 tok/s regardless of prefill length.

Behavior note: tone/"feel" differences vs a reference are **runtime config** (system prompt +
sampling), not the quant — a faithful int4 copy matches the reference once system prompt and
top-k / top-p / temperature match.

---

## On quantization support tables

Don't trust a fixed "what quantization compiles" matrix, including any you find in older copies
of this repo. Per-quant support is **compiler-build-dependent**: a result from one toolchain
build does not generalize to another, and rows that read "unsupported" have repeatedly turned out
to be one build's behaviour rather than a hardware or format limit.

Retest against your own target compiler rather than trusting a table.

---

## Runtime requirement (brief)

int4 AoT DGCs require a compatible GoogleTensor dispatch library to load on-device; a stock app's
dispatch may reject them as a firmware mismatch.