# Training and distilling a Nabra 7M How the small Nabra models are made, and the things that went wrong often enough to be worth writing down. The code that produced them is in `training/`. The 7M is a **distilled copy of an 82M**, not a separately trained model. The teacher emits the per-phoneme durations that generated its own audio and the student trains against those, which is why an 11x smaller network can follow it at all. --- ## Pipeline ``` corpus (audio + transcripts) -> manifest with phoneme ids build_sv2.py -> CTC aligner trained on THIS corpus train_aligner.py -> forced alignment -> durations align_ctc.py (+ pick_shift.py, resegment.py) -> GATE: offset sweep + reversed control gate16.py <-- do not skip -> teacher 82M train_student.py --config config_82m.json -> student 7M distilled from teacher distill_saudi.sh ``` ### 1. Manifest `build_sv2.py` walks a corpus and writes `{wav, text, ids, frames}` per clip, where `ids` are phoneme ids in the model's vocab. Frames are the student's own 40 Hz grid: a duration written here means exactly what the trainer will read. ### 2. Aligner Train the CTC aligner **on the target speaker's own audio**, not a general one. Durations transferred from a mismatched source are the single most expensive failure in this project (see Lessons, DTW). ```bash python train_aligner.py --manifest data/corpus.jsonl --max-frames 1000 --steps 24000 \ --out runs/aligner_x ``` ### 3. Align, and then prove the alignment is real ```bash SHIFT=$(python pick_shift.py data/index_probe.jsonl config_82m.json) python align_ctc.py --ckpt runs/aligner_x/last.pt --manifest data/corpus.jsonl \ --max-frames 1000 --shift "$SHIFT" --seg-max 800 --seg-min 60 \ --out data/index_raw.jsonl python resegment.py data/index_raw.jsonl data/index.jsonl python gate16.py data/index.jsonl config_82m.json 600 # <-- the gate ``` `gate16.py` reports two numbers and **both** must look right: | check | good | meaning | |---|---|---| | offset sweep | peak at `+0` | labels are not systematically early or late | | true labels | high dB gap | vowels are louder than pauses where the labels say so | | reversed control | near zero | the gap is not an artifact of any monotonic path | A recent Saudi run: sweep peaked at `+0`, true gap **+8.92 dB**, reversed control **+1.07 dB**. If the reversed control is also high, the alignment is a plausible-looking path, not a real one, and training on it will quietly produce a worse model. ### 4. Teacher, then student ```bash # teacher python train_student.py --steps 40000 --batch 8 --lr 2e-5 --lr-decay \ --mel-weight 5 --sil-weight 100 --slm-weight 1 --slm-adv-weight 0.2 \ --init-from nabra82m.pth --init-loose \ --index data/index.jsonl --config config_82m.json --out runs/teacher --resume # student: see distill_saudi.sh -- it waits for the teacher, evaluates it, dumps teacher # durations, and only then trains the 7M against them. ./distill_saudi.sh ``` `--init-loose` does a shape-matched partial load and **prints the percentage loaded plus the list of tensors left random**. Read that line. A checkpoint that silently loads 0% trains from scratch and looks like a slow run rather than a broken one. --- ## Lessons learned ### Alignment - **Never trust an alignment without the reversed-label control.** An MFCC+DTW aligner once placed every label 2 frames late because its frames were centred at `j*hop + n_fft/2` rather than `j*hop`. Training on it tripled WER. The offset sweep alone would not have caught it; the sweep plus the reversed control did. - **Measure the frame offset, do not assume it.** Different corpora in this line needed `+0` and `+1`. `pick_shift.py` measures it. - **Probes on 5k aligner steps are unreliable.** A 5k probe once preferred MSA phonemes over Egyptian (0.8525 vs 0.8541) and the full 24k run reversed the verdict decisively (0.2527 vs 0.2591). Run the probe long enough or do not run it. ### Evaluation - **UTMOS and WER disagree, and the disagreement is the information.** UTMOS rates how audio *sounds*; it cannot tell that the model said the wrong word. One 82M scored the best UTMOS of its group with nearly double the word error of the model beside it. - **Measure the corpus ceiling before judging a model.** Real audio from one Egyptian corpus scores UTMOS 3.356. A base model already scoring 3.774 cannot be improved by fine-tuning on it -- training pulls the model *down* toward the data. Always score the training audio. - **Small eval sets quantise.** At 38 reference words, one error is 0.026 WER and distinct systems land on identical numbers. Use hundreds of words, and repeat sampling models several times: one model spanned WER 0.079-0.158 across four runs on *identical* input. ### Training - **More steps is usually worse, past a point.** Repeatedly in this project the best checkpoint was mid-run: 45k beat 90k, 6k beat 12k, 3k beat 6k. Training loss kept falling the whole time. Snapshot often and pick on the curve; do not take the endpoint. - **A gentle fine-tune preserves the prior you are trying to replace.** Sweeping 1e-5 / 5e-5 / 7.5e-5 on the same corpus, 5e-5 won and 7.5e-5 overshot to a worse loss. - **Synthetic data costs the small model more than the large one.** Adding 13.4 h of synthetic speech (UTMOS 2.907) to 146 h of real audio improved the 82M's word error on the new domain and made the 7M worse on both axes. Capacity absorbs noisy data; 7M does not. - **Length distribution is a training variable.** A 30 s cap left only 8% of clips as long as the evaluation passage, and models trained that way ran away on long input -- one produced 120 s of audio for a 20 s passage. Lifting the cap to 60 s took long clips to 15% and fixed it. ### Text - **Use dialect text as written. Do not run an MSA diacritizer over it.** Measured on a Saudi corpus, MSA diacritization was the worst of three options: it imposes case endings and vowel patterns a dialect speaker does not pronounce. Keeping whatever tashkeel the transcript already carries beat both stripping and adding. - **Watch the ALBERT position limit.** `max_position_embeddings=512` is a hard cap on phoneme tokens. Two runs died at 537 and 569 tokens. `resegment.py` splits at aligner-found pauses so a segment never starts mid-word. - **Check vocabulary coverage before training, not after.** Count how many corpus characters are absent from the model's vocab. A corpus that is 99.997% covered may still carry invisible characters (Arabic Letter Mark, BOM, LAM-ALEF ligatures) that become unknown tokens; NFKC plus a strip pass fixes it. ### Packaging - **`student.pth` is not directly loadable.** The trainer writes a flat state dict (`bert.x`, `decoder.y`); KModel wants it nested per block (`{"bert": {...}}`). Publishing the flat file fails with `AssertionError: bert.embeddings.word_embeddings.weight`. See `nest()` in `eval_student.py`. - **The voicepack is part of the model.** Evaluating with `af_heart` a model trained with `af_msa` cost UTMOS 4.138 -> 3.570 and WER 0.0525 -> 0.0701. Ship the pack you trained with and default to it everywhere. - **Kokoro hardcodes the decoder widths.** The stock package raises `TypeError` on configs that set `hidden_channels`/`out_channels`, which is why a patched copy is vendored with every release. ### Operations - **Bracket your `pkill` patterns.** `pkill -f "train_student.py"` matches the shell running it. Write `train_studen[t].py`. - **Deleting an `hf_hub_download` path frees nothing.** It returns a symlink into `snapshots/`; the real file is in `blobs/`. A streaming job that removed only the symlink silently kept 23 GB.