gcoli/whisper-large-v3-turbo-swiss-german-mit

Whisper Large-v3-Turbo Swiss German — commercially usable (MIT)

Swiss German speech to Standard German text. LoRA fine-tune of openai/whisper-large-v3-turbo, merged into the weights, with a deliberately clean licence chain: MIT base model, MIT training data, MIT result.

Most public Swiss German models inherit CC BY-NC from SwissDial and cannot be used commercially. This checkpoint avoids that: it is trained only on MIT-licensed FHNW/i4ds material.

Results

Measured on the FHNW/i4ds All Swiss German Dialects Test Set (MIT, 5750 clips, 12.7 h, 17 dialects), decoded with faster-whisper (CTranslate2, float16, beam 5) on an RTX 3080 Ti.

Model	WER dev	WER final	CER	RTF
this checkpoint	23.7 %	25.9 %	13.4 %	0.033
openai/whisper-large-v3-turbo (no fine-tune)	26.5 %	28.5 %	14.6 %	0.036
openai/whisper-large-v3 (no fine-tune)	24.0 %	26.3 %	13.7 %	0.080

2.8 WER points better than the untouched base model at identical speed.

The test set is split speaker-disjointly into two halves: dev (2930 clips) was used for model selection, final (2820 clips) was held out. The final half is harder for every model measured, including untrained baselines — compare within a column, not across. The test set was never trained on.

WER/CER are per-clip means after normalisation (lowercase, punctuation removed, ß → ss).

Usage

faster-whisper / CTranslate2 — weights are in the ct2/ subfolder, so fetch that folder first (the repository root holds the transformers format, whose config.json differs):

from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel

path = snapshot_download("gcoli/whisper-large-v3-turbo-swiss-german-mit",
                         allow_patterns="ct2/*")
model = WhisperModel(f"{path}/ct2", device="cuda", compute_type="float16")
segments, info = model.transcribe("input.wav", language="de", beam_size=5)
print(" ".join(s.text.strip() for s in segments))

transformers — the repository root:

from transformers import pipeline

asr = pipeline("automatic-speech-recognition",
               model="gcoli/whisper-large-v3-turbo-swiss-german-mit", device=0)
print(asr("input.wav", generate_kwargs={"language": "german", "task": "transcribe"})["text"])

Use language token de and task transcribe: the model outputs Standard German, not dialect spelling. Audio should be 16 kHz mono.

Long recordings

On recordings of several minutes, turn context carry-over off. faster-whisper leaves condition_on_previous_text on by default, and on 15-minute files it cost this model family 2.5 (turbo) to 11.7 (large) WER points in our measurements, with the output growing ~14 % longer than the reference — the model starts inventing text. VAD filtering did not help either.

segments, info = model.transcribe("meeting.wav", language="de", beam_size=5,
                                  condition_on_previous_text=False, vad_filter=False)

Measured on 26 files of ~15 min each, built by concatenating the held-out half of the test set, so the audio content is identical to the short-form numbers above. Expect roughly 1 to 3 WER points worse on long recordings than on single sentences; that gap differs per model and does not follow the short-form ranking.

Compatibility

• Format: safetensors (fp16) at the repository root, CTranslate2 float16 under ct2/
• Runtimes: transformers, faster-whisper / CTranslate2
• int8_float16 quantisation costs about 0.3 WER points on this checkpoint's base family and is a reasonable trade for VRAM-constrained setups
• For MLX (Apple Silicon), convert the root safetensors with mlx_whisper

Training

Base	openai/whisper-large-v3-turbo (MIT)
Method	LoRA r=32, α=64, target q_proj/v_proj, dropout 0.05, merged after training
Optimiser	lr 3e-4, linear decay, 500 warmup steps, 1 epoch, batch 16, bf16 + gradient checkpointing
Data	40 h FHNW/i4ds Swiss Parliaments Corpus v2 (Bern) + 40 h Gemeinderat Zürich
Hardware	single RTX 3080 Ti (12 GB), ~1.6 h training

The Zürich corpus is audio only. Transcripts were generated with openai/whisper-large-v3 (Apache-2.0) and filtered by confidence (avg_logprob > -0.45, no_speech_prob < 0.3, compression ratio < 2.2, repetition-loop rejection).

Two findings from the run series this checkpoint comes from:

• Learning rate dominates. The same recipe at lr 1e-3 produced no gain at all over the base model; 3e-4 gained 1.8 points immediately.
• Source mixture matters more than volume. Both corpora must be present — Bern-only and Zürich-only fine-tunes are clearly worse — but the ratio between 50 % and 89 % Zürich is not measurably different, and 80 h performs as well as 160 h or 247 h.

Other languages

Fine-tuning on one language variety can damage the others. Measured on google/fleurs (250 validation clips per language, no prompt, beam 5):

	German	English	French	Italian	Spanish
this checkpoint	5.2 %	5.7 %	5.5 %	3.4 %	3.9 %
openai/whisper-large-v3-turbo	4.9 %	4.9 %	5.2 %	2.4 %	3.2 %

The fine-tune costs 0.3 to 1.0 WER points in the other languages — a real but mild regression, not catastrophic forgetting. If you need a multilingual system, keep the base model for everything except Swiss German.

Limitations

• The test set is read speech; the training data is parliamentary speech. Performance on spontaneous conversation, podcasts or noisy recordings is unmeasured.
• Training data covers two regions only (Bern, Zürich). Valais and Fribourg remain the weakest dialects at roughly 30–35 % WER.
• Zürich transcripts are pseudo-labels: the student cannot substantially exceed its teacher.
• Whisper can hallucinate or omit text, particularly with noise, silence, overlapping speakers, uncommon dialects or specialised vocabulary. Do not use its output as the sole basis for high-impact decisions. Obtain consent before transcribing people.

Provenance

• Base: openai/whisper-large-v3-turbo
• Pseudo-label teacher: openai/whisper-large-v3 (Apache-2.0)
• Training data: FHNW/i4ds Swiss Parliaments Corpus v2 and Gemeinderat Zürich Audio Corpus, both MIT, https://www.cs.technik.fhnw.ch/i4ds-datasets
• Evaluation: FHNW/i4ds All Swiss German Dialects Test Set (MIT), same source
• Details in training_provenance.json

License and usage conditions

MIT. The base model (openai/whisper-large-v3-turbo) is MIT, the training data (FHNW/i4ds Swiss Parliaments, Gemeinderat Zürich) is MIT, and the pseudo-label teacher (openai/whisper-large-v3) is Apache-2.0. Commercial use is permitted.

Please credit FHNW/i4ds for the underlying corpora. This repository grants no additional rights to upstream models, training data or software, and implies no endorsement by FHNW, i4ds or OpenAI.