WRITEUP.md

# Distilling an operator-video quality stack into a ~2 MB on-device model

This is the design write-up for `video_benchmark.distill` — how a heavy, multi-model
video-quality scoring stack is compressed into a **single ~2.5 M-parameter network**
that reproduces it in **one forward pass** and ships at **~2 MB** on device (candle /
Rust and Apple MLX), while staying honest about what it does and does not learn.

---

## 1. The problem

The production pipeline scores first-person / operator video frames (headband
cameras, robotics teleop) on several quality axes. The accurate version runs a stack
of models per frame:

- **Classical OpenCV metrics** — brightness, sharpness, blur, exposure anomalies.
- **A 3-paradigm learned-IQA ensemble** (via `pyiqa`): **TOPIQ** (CNN), **MUSIQ**
  (ViT) and **CLIP-IQA+** (CLIP) — three different inductive biases voting on
  perceptual quality.
- **A MobileCLIP zero-shot scene classifier** — is this a usable operator scene?

That is great for an offline batch, but it is several hundred milliseconds per frame
and many hundreds of MB of weights — a non-starter for on-device / real-time use.

**Goal:** one compact model that emits all eight signals at once, fast, small enough
to embed, and faithful enough to trust.

---

## 2. The teacher → student design

```
                 ┌─────────────── teacher stack (label generator) ───────────────┐
   frame  ─┬───► OpenCV: brightness · sharpness · blur · anomaly                  │
           ├───► pyiqa ensemble: TOPIQ(CNN) + MUSIQ(ViT) + CLIP-IQA+(CLIP)        │
           └───► MobileCLIP zero-shot scene usability                            │
                 └──────────────────────────► 8 per-frame targets (0..100) ──────┘
                                                          │  (distillation labels)
   frame  ──► frozen tiny backbone ──► embedding ─┐       ▼
                                                  ├──► fuse ──► residual-MLP trunk ──► 8 MLP heads ──► 8 scores
   frame  ──► 10 classical descriptors ───────────┘                                   (100·σ, bounded 0..100)
```

The **student** is `CompactQualityNet` (`model.py`):

- **Frozen backbone** — a `tiny` preset MobileNetV3-Small (≈2.45 M params). Frozen so
  the embedding can be **precomputed once** and every training epoch is a cheap matrix
  op on the cache (`data.py`). Presets: `tiny` (default), `micro` (LCNet-050, sub-2 MB
  at every precision), `clip` (the legacy MobileCLIP-S0, max fidelity).
- **Descriptor fusion** (see §4) — the embedding is concatenated with 10 cheap
  classical descriptors before the trunk.
- **Residual-MLP trunk** — pre-norm residual blocks (`x + W₂·GELU(W₁·LN(x))`).
- **Per-target heads** — one small 2-layer MLP per signal, so each stays independently
  calibrated while sharing the trunk.
- **Bounded output** — `100·sigmoid(logit)`: scores start neutral at 50 and can never
  go negative or saturate the clamp, which keeps them calibrated.

The architecture (trunk dim, blocks, head dim, `extra_dim`) is **persisted in the
checkpoint**, so `infer.py` rebuilds the exact network even if defaults later change.

---

## 3. Training (`train.py`)

A real mini-batch loop, not a single least-squares step:

- **kornia augmentation** (`augment.py`) — each sampled frame is expanded into several
  views: a **log-uniform Gaussian-blur** sweep (perceptual blur is multiplicative, so
  log-uniform spreads samples evenly across mild→heavy), a mild **motion blur** (camera
  shake), plus colour/gamma jitter and sensor noise. View 0 is the untouched original.
  This multiplies the data and injects spread into otherwise near-flat signals so their
  distilled correlations become meaningful instead of `n/a`. Empirically, this lifts the
  rank-correlation metrics; *aggressive* degradations (strong motion blur, JPEG
  blocking) were measured to regress fidelity on this small corpus and are left out.
- **SGDR** — `CosineAnnealingWarmRestarts`: the LR periodically "resurges" to escape
  plateaus.
- **EMA** — an exponential-moving-average shadow of the trainable weights for a
  smoother final model (LayerNorm-only trunk, so no BatchNorm running-stat hazard).
- **Loss** — `SmoothL1(β=0.1)` (β=1.0 degenerates to plain MSE on [0,1] targets) **plus
  a differentiable Pearson-correlation term**, so we optimise the exact fidelity
  metric (PLCC) the evaluation reports. Per-target weighting counts the deep signals
  double.
- **Ship-by-deep-PLCC** — on tiny data the EMA is not guaranteed to win, so we keep the
  best EMA and the best raw weights and ship whichever scores higher held-out
  deep-PLCC.
- **`accelerate`** — the same loop is correct on CPU / Apple MPS / CUDA.

---

## 4. The key idea: classical-descriptor fusion (`descriptors.py`)

A frozen ImageNet backbone is *trained to be invariant* to exactly what some quality
signals measure: it normalises away absolute exposure, global contrast and colour
cast. So `brightness` and exposure `anomaly` are nearly unrecoverable from its
embedding — their distilled correlation collapsed to ~0.

**Fix:** a hybrid hand-crafted + deep representation. We concatenate a fixed 14-D
vector of normalised classical statistics to the embedding:

- **Photometric / focus (10):** luma mean/std, dark/bright fraction, RMS contrast,
  colourfulness (Hasler–Süsstrunk), saturation, Laplacian-variance sharpness, Canny
  edge density, Tenengrad.
- **BRISQUE natural-scene statistics (4):** on the MSCN field
  `I = (luma − μ_local) / (σ_local + 1)`, a pristine image's coefficients are
  ~unit-Gaussian; blur/noise/compression push the **variance, excess kurtosis** and the
  **horizontal/vertical neighbour-product means** off their natural values — a strong,
  classic no-reference quality cue (Mittal et al., 2012).

All are computed **identically** at cache-build and inference time with fixed scales
(no data-dependent statistics), so the model stays self-contained and reproducible.

Two design choices make this strictly beneficial (`model._FusedInput`):

1. **Split projection.** The embedding keeps its own projection; the descriptors enter
   through a **separate** projection that is added on. This keeps the big embedding
   matrix's row length **block-aligned**, so it int4/int8 block-quantises in GGUF/MLX
   (a single fat `Linear` over the concatenation would have an unaligned row and fall
   back to fp16 — ~4× larger; this is exactly the regression that pushed an early build
   to 2.38 MB).
2. **Zero-initialised.** The descriptor projection starts at zero, so fusion is an
   exact **no-op at init** and can only *help* — descriptors are used only insofar as
   they reduce the loss, never diluting the embedding the deep signals rely on.

Result: best composite scores of any build, `clipiqa`/`anomaly`/`brightness` lifted,
deep-signal PLCC held — at negligible size cost.

---

## 5. Evaluation (`evaluate.py`)

IQA-grade reporting, with rigor appropriate to a small held-out set:

- **The standard quad per signal** — PLCC (Pearson), SRCC (Spearman), **KRCC
  (Kendall)**, plus **MAE and RMSE** (0–100 units).
- **VQEG logistic-fitted PLCC / RMSE** — the IQA convention (ITU-T P.1401 / VQEG) maps
  predictions through a monotonic **5-parameter logistic** before correlating, so a
  model isn't penalised for an arbitrary nonlinear score scale when its ranking is
  right. Reported next to the raw numbers.
- **Composite** — agreement on the overall quality verdict (mean over signals that
  genuinely vary), the bottom-line number.
- **Deep-signal** aggregates — mean over the signals a learned model actually earns
  (`iqa, musiq, clipiqa, scene`).
- **BCa bootstrap 95 % CI** on the composite PLCC — a single estimate on ~190 frames
  hides real uncertainty. We use the **bias-corrected-and-accelerated** bootstrap
  (Efron, 1987), which corrects the percentile interval for the median-bias and skew a
  correlation's sampling distribution has near ±1; it degrades to a percentile
  bootstrap if BCa can't be formed.
- **Robustness sweep** — blur frames at increasing σ and check the student *tracks* the
  teacher's degradation (not just degrades arbitrarily), reported as a tracking MAE.
- **Fidelity vs σ** — deep-PLCC bucketed by blur severity: a clean frame is easy, heavy
  blur is the real test.
- **Honesty guards** — a near-flat teacher signal (std < 5 on 0–100) is reported as
  `n/a`, never a fake correlation. The train/val split is **leakage-free by clip**.

All of this is rendered to `report.md` / `report.txt` and published to the Hub in the
modern **`.eval_results/`** format (PLCC/SRCC/KRCC/MAE/RMSE per signal + composites),
keyed to a companion benchmark dataset.

---

## 6. Results (latest run)

Final `tiny` + fusion run on the operator-video corpus (see `report.md` for the full
per-signal table with KRCC/RMSE and the blur sweep):

| metric | value |
|---|---|
| **Composite PLCC** | **0.899** (95 % BCa CI [0.854, 0.925]) |
| Composite PLCC — VQEG logistic-fitted | 0.902 |
| Composite SRCC / KRCC | 0.896 / 0.714 |
| Deep-signal PLCC | 0.69 |
| `iqa` / `musiq` PLCC | 0.88 / 0.84 |
| `sharpness` / `blur` PLCC | 0.90 / 0.86 |
| Throughput | ~120× the teacher stack |
| Params / int4 weights | 2.45 M / ~1.9 MB |
| int4 GGUF / MLX file | 2.00 MB / 2.40 MB |

---

## 7. Deployment (`quantize.py`, `export_hf.py`)

Sub-fp16 packing with **real, scale-aware size accounting** (it models the fp16
fallback for tensors GGUF can't block-quantise, so the reported int4/int8 numbers
match what the files actually weigh — not an optimistic `params × bytes` guess):

- **GGUF** (candle / Rust) — ggml Q8_0 / Q4_0 + fp16 fallback. Arch + target order +
  descriptor names ride along as metadata so a candle program rebuilds the forward
  pass. int4 GGUF ≈ 2.0 MB.
- **MLX** (Apple Silicon) — grouped affine quant, int8 + int4, at the empirically
  smallest group size per bit-width (smaller groups qualify more matrices and cut fp16
  fallbacks). Verified with an MLX-native quantise→dequantise round-trip self-test.
- **safetensors / `.pt`** — the canonical fp16 weights `infer.py` loads.
- **LICENSE / NOTICE** — backbone-aware: MIT for the trained trunk + heads; the timm
  MobileNetV3 backbone is Apache-2.0 (apple-amlr only when a MobileCLIP tower is
  actually bundled); teacher models (BSD pyiqa, apple-amlr MobileCLIP) are used only to
  generate labels and are **not** redistributed.

---

## 8. Reproduce

```bash
uv sync --group distill                 # gguf + mlx export backends (optional)
uv run python -m video_benchmark.distill --videos videos --epochs 300 \
    --preset tiny --quantize int4 --gguf --mlx --fusion
uv run python -m video_benchmark.distill.infer frame.jpg   # run the distilled model
uv run pytest tests/test_distill.py tests/test_distill_model.py
```

---

## 9. Honest limitations

- **`scene` is flat** on this corpus (only a handful of clips, all valid operator
  scenes), so its distilled correlation is `n/a` — a *data* limitation, not a model
  one. More diverse footage would give it real spread.
- **`brightness` / `anomaly`** are exact OpenCV stats in production; the distilled head
  is only a unified read-out for them. Fusion lifts them but they are better computed
  directly.
- **The Hub benchmark** uploads correctly, but HF only renders a *live* leaderboard
  once it allow-lists the `eval.yaml` — and no `evaluation_framework` enum value fits a
  custom distillation-fidelity eval, so that field is a documented placeholder.

---

## 10. References

The design borrows established methods rather than inventing them:

- **Knowledge distillation** — Hinton, Vinyals & Dean, *Distilling the Knowledge in a
  Neural Network*, 2015.
- **BRISQUE / MSCN natural-scene statistics** — Mittal, Moorthy & Bovik, *No-Reference
  Image Quality Assessment in the Spatial Domain*, IEEE TIP 2012.
- **VQEG logistic fit for IQA** — VQEG, *Final Report on the Validation of Objective
  Models of Video Quality Assessment*, 2003; ITU-T Rec. P.1401.
- **BCa bootstrap** — Efron, *Better Bootstrap Confidence Intervals*, JASA 1987.
- **SGDR warm restarts** — Loshchilov & Hutter, *SGDR: Stochastic Gradient Descent with
  Warm Restarts*, ICLR 2017.
- **Weight averaging (EMA / Polyak)** — Polyak & Juditsky, 1992; Izmailov et al., *SWA*,
  2018.
- **Backbone & teachers** — MobileNetV3 (Howard et al., 2019); MobileCLIP (Vasu et al.,
  CVPR 2024); the learned-IQA ensemble TOPIQ (Chen et al., 2024), MUSIQ (Ke et al.,
  ICCV 2021), CLIP-IQA (Wang et al., AAAI 2023), via `pyiqa`.
- **Colourfulness** — Hasler & Süsstrunk, *Measuring Colourfulness in Natural Images*,
  2003. **Tenengrad focus** — Krotkov, 1987.
- **kornia** — Riba et al., *Kornia: an Open Source Differentiable Computer Vision
  Library for PyTorch*, WACV 2020.
- **GGUF / ggml k-quants** (llama.cpp) and **MLX** (Apple) for the on-device exports.