whisper-small-waxal-matched-lug (matched-recipe Luganda ablation)

Research ablation, not the recommended Luganda model. This is Whisper-Small fine-tuned on Luganda only, with exactly the recipe of the leave-one-out model whisper-small-waxal-loo-lug. It exists so that "trained on Luganda" and "trained on the 18 other languages" are compared under one recipe. For Luganda ASR use whisper-small-waxal-lug.

Result (Luganda test split)

Setting	WER	CER
Leave-Luganda-out (18 other languages, same recipe)	86.4	32.0
This model (Luganda only, same recipe)	283.0	158.0
Released per-language Whisper-Small (benchmark recipe)	21.6	n/a

Test utterances scored: 638.

Known failure: language drift. Luganda is not a Whisper language and this recipe decodes without a language token, so Whisper's language detection often falls back to English. With only 5,455 Luganda training utterances, fine-tuning did not override that prior: 124 of 638 test utterances become English, repetitive output more than 3x the reference length, which drives WER above 100 (112.6 WER on the remaining utterances). Retraining for 1,400 steps (epoch-matched to the Ewe and Fula runs) made this worse (192/638 runaway, 373.6 WER), so it is not overfitting. The 18-language leave-one-out model sees about 54k utterances without a language token and does not show this failure.

Training

All Luganda utterances in the WAXAL train split (google/WaxalNLP), 16 kHz mono; transcripts NFC-normalized, lower-cased, punctuation removed, diacritics preserved. Labels longer than Whisper's 448-token limit are truncated.

Hyperparameter	Value
Base	`openai/whisper-small` (244M)
Training languages	1 (Luganda)
Steps	4,000
Optimizer	AdamW, lr 1e-5, 200 warmup steps
Batch size	16
Precision	fp16
Language token	none (same as the leave-one-out model)
Hardware	1x NVIDIA H200

Evaluation

Greedy decoding on the full Luganda WAXAL test split; WER and CER with jiwer on NFC-normalized, lower-cased text with diacritics preserved.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import soundfile as sf

model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")
processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")
wav, sr = sf.read("your_audio.wav")            # 16 kHz mono
feats = processor(wav, sampling_rate=16000, return_tensors="pt").input_features
ids = model.generate(feats)
print(processor.batch_decode(ids, skip_special_tokens=True))

Intended and prohibited uses

Intended use. Research on, and development of, speech recognition for the language(s) this model was trained on, in particular for low-resource and under-served language communities. Error rates are still too high for high-stakes uses such as healthcare, legal, or emergency settings without further validation on the target domain.

Prohibited uses. This model must not be used for non-consensual applications, including surveillance or monitoring of individuals or communities, covert recording or transcription of people without their consent, or voice cloning and impersonation. Please respect the consent and privacy of the speakers whose speech you process.

Citation

Part of the WAXAL ASR Benchmark (arXiv:2606.02375).

@article{waxalnet2026,
  title  = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages},
  author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others},
  year   = {2026},
  note   = {arXiv preprint arXiv:2606.02375}
}