Robust Speech Recognition via Large-Scale Weak Supervision
Paper • 2212.04356 • Published • 55
whisper-large-v3-turbo-danish
░▒▓█╔════════════════════════════════════════════════════════════════════╗█▓▒░
░▒▓█│ │█▓▒░
░▒▓█│ ██╗ ██╗██╗ ██╗██╗███████╗██████╗ ███████╗██████╗ │█▓▒░
░▒▓█│ ██║ ██║██║ ██║██║██╔════╝██╔══██╗██╔════╝██╔══██╗ │█▓▒░
░▒▓█│ ██║ █╗ ██║███████║██║███████╗██████╔╝█████╗ ██████╔╝ │█▓▒░
░▒▓█│ ██║███╗██║██╔══██║██║╚════██║██╔═══╝ ██╔══╝ ██╔══██╗ │█▓▒░
░▒▓█│ ╚███╔███╔╝██║ ██║██║███████║██║ ███████╗██║ ██║ │█▓▒░
░▒▓█│ ╚══╝╚══╝ ╚═╝ ╚═╝╚═╝╚══════╝╚═╝ ╚══════╝╚═╝ ╚═╝ │█▓▒░
░▒▓█│ │█▓▒░
░▒▓█│ ░░▒▒▓▓ D A N I S H S P E E C H ▓▓▒▒░░ │█▓▒░
░▒▓█│ │█▓▒░
░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
░▒▓█│ R E L E A S E I N F O [ MODEL / WEIGHTS ] │█▓▒░
░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
░▒▓█│ │█▓▒░
░▒▓█│ EDITION .......... LARGE-V3-TURBO │█▓▒░
░▒▓█│ LANGUAGE ......... DANISH │█▓▒░
░▒▓█│ STATUS ........... SOTA-ish │█▓▒░
░▒▓█│ LAYERS ........... 32 ENCODER / 4 DECODER │█▓▒░
░▒▓█│ MODEL WIDTH ...... 1280 │█▓▒░
░▒▓█│ AUDIO INPUT ...... 16 kHz MONO / 128 MEL BINS │█▓▒░
░▒▓█│ TRAIN WINDOW ..... 30 SECONDS │█▓▒░
░▒▓█│ VOCABULARY ....... 51,866 TOKENS │█▓▒░
░▒▓█│ OPTIMIZER ........ MUON + ADAMW │█▓▒░
░▒▓█│ TRAINING ......... 200,000 OPTIMIZER STEPS │█▓▒░
░▒▓█│ TRAINED ON ....... NVIDIA GEFORCE RTX 4090 │█▓▒░
░▒▓█│ │█▓▒░
░▒▓█├────────────────────────────────────────────────────────────────────┤█▓▒░
░▒▓█│ 01 USAGE / 02 TRAINING / 03 EVALUATION │█▓▒░
░▒▓█│ 04 LIMITATIONS / 05 LICENSE / 06 CREDITS │█▓▒░
░▒▓█│ │█▓▒░
░▒▓█╚════════════════════════════════════════════════════════════════════╝█▓▒░
░░▒▒▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▒▒░░
This is a Danish continuation of
openai/whisper-large-v3-turbo.
The architecture and tokenizer are unchanged. The released weights are an
exponential moving average taken after 200,000 optimizer steps.
01 / USAGE
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
model_id = "thorh/whisper-large-v3-turbo-danish"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(
model_id,
dtype=dtype,
low_cpu_mem_usage=True,
).to(device)
transcribe = pipeline(
"automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
dtype=dtype,
device=device,
)
result = transcribe(
"audio.wav",
generate_kwargs={"language": "da", "task": "transcribe", "num_beams": 1},
)
print(result["text"])
02 / TRAINING
Training used the Danish portions of CoRal v3, Common Voice 26.0, FLEURS,
FTSpeech and NST together with filtered pseudo-labelled data. The pseudo-labels
were generated with openai/whisper-large-v3
and confidence-filtered before use.
Targets retained their original casing, punctuation and numeral style. Audio augmentation, SpecAugment and synthetic no-speech examples were included. The data design was inspired by the weak-supervision and no-speech principles in the original Whisper paper.
| Setting | Value |
|---|---|
| Steps | 200,000 |
| Optimizer | Muon for hidden matrices; AdamW for remaining parameters |
| Peak learning rate | Muon 5e-6; AdamW 2.5e-6 |
| Weight decay | 0.01 |
| Schedule | 1,000-step warmup, stable to 50,000, linear decay to 10% at 200,000 |
| EMA | 0.9999 per step |
| Precision | BF16 parameters, FP8 hidden forward GEMMs, compensated FP32 optimizer updates |
| Maximum training segment | 30 seconds |
| Training hardware | One NVIDIA GeForce RTX 4090 |
| Active wall time | Approximately 56 hours |
The following figures are cumulative audio sampled during training. They include repeated sampling and are not unique dataset durations.
| Source | Hours seen |
|---|---|
| CoRal conversation | 485.45 |
| CoRal read-aloud | 894.45 |
| Common Voice Danish | 72.63 |
| FLEURS Danish | 74.06 |
| FTSpeech | 1,469.10 |
| NST Danish | 712.01 |
| Pseudo-labelled data | 9,200.04 |
| Total | 12,907.73 |
Additional recipe details are recorded in training_recipe.json.
03 / EVALUATION
The checkpoint was evaluated against the public Danish ASR Leaderboard protocol using greedy Danish transcription. Short clips used standard decoding; clips over 30 seconds used timestamped sequential long-form decoding. The leaderboard normalizer was applied, including Danish numeral normalization.
| Test set | Samples | WER (%) | CER (%) |
|---|---|---|---|
| CoRal conversation | 8,438 | 20.05 | 11.92 |
| CoRal read-aloud | 9,122 | 12.93 | 4.80 |
Common Voice (cv17_da) |
2,756 | 8.24 | 2.65 |
FLEURS da_dk |
930 | 8.95 | 3.65 |
FTSpeech test_balanced |
5,534 | 7.39 | 4.19 |
| Unweighted domain mean | 26,780 | 11.51 | 5.44 |
Exact revisions and decoding settings are recorded in training_recipe.json.
04 / LIMITATIONS
- Accuracy in languages other than Danish may be reduced.
- Long-form behavior was not directly trained and should be validated separately.
05 / LICENSE
The model weights are released under the MIT License. Training data are not
redistributed and remain subject to their source terms. See
THIRD_PARTY_NOTICES.md.
06 / CREDITS
@article{radford2022robust,
title={Robust Speech Recognition via Large-Scale Weak Supervision},
author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
journal={arXiv preprint arXiv:2212.04356},
year={2022}
}
- Downloads last month
- 490