Hugging Face's logo

Loudness-ai
/
urdu-sukhan-asr-v1

Automatic Speech Recognition
Transformers
Safetensors
Urdu
English
whisper
urdu
speech-recognition
asr
speech-to-text
code-switching
whisper.cpp
ggml
ctranslate2
faster-whisper
loudness-ai
Eval Results (legacy)
Model card Files
xet
 Community
1

Request access to Urdu Sukhan ASR v1

Loudness.ai reviews every request by hand.

Urdu Sukhan ASR v1 is released under the CC BY-NC 4.0 licence, for non-commercial use only. For commercial use, contact us at https://loudness.ai/contact-us/. When you request access, Hugging Face shares your username and email address, and your answers below, with Loudness.ai. We use them only to review your request and to contact you about it.

Log in or Sign Up to review the conditions and access this model content.

Urdu Sukhan ASR v1

Urdu Sukhan ASR is an Urdu speech-to-text model by Loudness.ai. It also transcribes English and speech that switches between Urdu and English, keeping English words in English letters. It is a full fine-tune of OpenAI's Whisper large-v3-turbo, so it keeps that model's size and speed (809 M parameters) and runs wherever Whisper runs: Python, whisper.cpp on CPUs, Macs and phones, and faster-whisper.

  • Fewer word errors than large-v3-turbo on every Urdu test set we have: 39% fewer on average over seven held-out sets, from 30.9% to 12.7% on IndicVoices and from 88.9% to 23.6% on one- or two-word replies.
  • Urdu–English code-switching: 19.7% word error rate on clips that switch from Urdu to English (base model 53.6%), with English unchanged at 4.9%.
  • Silence stays silent: with the setting below, it writes nothing on noise, music and silence.
  • Half the size of Whisper large-v3: 809 M parameters against 1.55 B; the q5_0 file is 574 MB against 1,081 MB (details under Speed).
  • Every runtime: Transformers, whisper.cpp (q8_0, q5_0, q4_0) and CTranslate2 in one repository.

Try it without installing anything: Open in Colab records your voice or takes a file and transcribes it on Colab's free GPU (after your access request is approved).

Know before you use it: names are its weakest point (11 of 23 spelled right in our test chapter); see Limitations and what the next version will fix.

On loudness.ai: technical report (LA-RB-2026-004) · archived on Zenodo (doi:10.5281/zenodo.23518703) · model page · install guide · en español: informe, modelo, guía.

Code, examples and the development log: github.com/Loudness-ai/urdu-sukhan-asr.

Quick start

A step-by-step guide for Windows, macOS and Linux is on loudness.ai.

Try it in Colab

Open in Colab: record or upload audio and transcribe it on Colab's free GPU, with your approved Hugging Face login.

Get access

The model is free for non-commercial use under CC BY-NC 4.0, and access is approved by hand: fill in the form at the top of this page and wait for approval. Then create a read token and log in once on each machine, so the downloads below can authenticate:

pip install -U huggingface_hub
hf auth login

Use Python 3.12 for the examples below (python3.12 -m venv .venv): some of their dependencies have no ready-made wheels for newer Python versions on macOS yet.

For commercial use, contact us.

Transformers (GPU, Apple GPU or CPU) 

pip install transformers torch soundfile

import torch
from transformers import pipeline

device = "cuda:0" if torch.cuda.is_available() else "cpu"
asr = pipeline("automatic-speech-recognition", model="Loudness-ai/urdu-sukhan-asr-v1",
               dtype=torch.float16 if device != "cpu" else torch.float32, device=device)
result = asr("speech.wav", chunk_length_s=30, batch_size=4,
             generate_kwargs={"language": "ur", "task": "transcribe"})
print(result["text"])

Passing a file name needs ffmpeg installed. chunk_length_s=30 lets it transcribe recordings of any length. For English set "language": "en"; leave the language out to detect it. A complete script that also reads FLAC and MP3 without ffmpeg is examples/transcribe_transformers.py.

faster-whisper (fastest for long recordings on a CPU) 

pip install faster-whisper huggingface_hub

from faster_whisper import BatchedInferencePipeline, WhisperModel
from huggingface_hub import snapshot_download

path = snapshot_download("Loudness-ai/urdu-sukhan-asr-v1", allow_patterns=["ctranslate2/*"]) + "/ctranslate2"
model = WhisperModel(path, device="cpu", compute_type="int8", cpu_threads=8)   # device="cuda", compute_type="float16" on a GPU
segments, _ = BatchedInferencePipeline(model).transcribe(
    "recording.wav", language="ur", beam_size=5, batch_size=8, vad_filter=True, suppress_blank=False,
    without_timestamps=True, condition_on_previous_text=False)
print(" ".join(s.text.strip() for s in segments))

The voice detector cuts the audio into speech pieces that are transcribed in batches. It can drop faint speech, such as quiet English after Urdu: set vad_filter=False to keep it (--no-vad in the full script); with suppress_blank=False, silence still gives no text. Full script with timing: examples/transcribe_faster_whisper.py.

whisper.cpp (CPUs, Apple Silicon, mobile) 

hf download Loudness-ai/urdu-sukhan-asr-v1 ggml/ggml-urdu-sukhan-asr-v1-q8_0.bin --local-dir .
whisper-cli -m ggml/ggml-urdu-sukhan-asr-v1-q8_0.bin -l ur -bs 5 -nt -mc 0 -f speech.wav

examples/whisper_cpp.sh builds whisper.cpp v1.9.5 with a one-line patch that adds an -nsb switch (silence gives no text, see below), downloads the model and the Silero voice detector, and transcribes a file with the recommended settings (VAD=0 turns the detector off, which keeps faint speech it can drop). In your own C/C++ code, set whisper_full_params.suppress_blank = false instead.

Files

Path Format Size Use
model.safetensors + configs Transformers, fp16 1.6 GB Python on GPUs and CPUs, further fine-tuning
ggml/ggml-urdu-sukhan-asr-v1-q8_0.bin whisper.cpp, 8-bit 874 MB best quality in whisper.cpp; Apple Silicon
ggml/ggml-urdu-sukhan-asr-v1-q5_0.bin whisper.cpp, 5-bit 574 MB smaller; slowest of the three on x86
ggml/ggml-urdu-sukhan-asr-v1-q4_0.bin whisper.cpp, 4-bit 474 MB fastest on x86 servers and smallest
ctranslate2/ CTranslate2, int8 824 MB faster-whisper

The original full-precision checkpoint was lost when our training server was shut down. The Transformers weights here were rebuilt from the q8_0 file, which stores each block of 32 weights as 8-bit values with one scale, so they are the q8_0 model in fp16. Re-run on three test sets, they score within 0.3 points of the original on Common Voice and the first audiobook and 0.7 points better on the second (VERIFY.md). A Core ML encoder of the full model is not included yet: in fp16 it did not match PyTorch closely enough.

Results

Word error rate (WER, %; lower is better), Transformers, greedy decoding, the same text normalisation for both models (punctuation, diacritics, digit forms, Arabic/Urdu letter forms and hamza variants made consistent). None of these clips were used in training.

Test set (clips) Whisper large-v3-turbo Urdu Sukhan ASR v1
IndicVoices Urdu, validation (300) 30.9 12.7
Common Voice Urdu, 15 speakers not seen in training (260) 39.8 28.1
FLEURS Urdu, test (299) 21.0 16.2
UrduSpeech, test split (100) 14.5 8.8
nzg73, held-out clips (300; same speaker as its training part) 4.8 1.6
Audiobook narrator 1, held-out chapters (48) 5.9 4.0
Audiobook narrator 2, held-out chapter (10) 17.2 13.2

Robustness ("stress") sets, same decoding:

Stress set (clips) Whisper large-v3-turbo Urdu Sukhan ASR v1
Quiet speech, 30 dB softer (40) 35.6 23.5
Phone-line speech (40) 36.2 25.0
One- or two-word replies (40) 88.9 23.6
Urdu then English in one clip (20) 53.6 19.7
Continuous speech, 25–30 s (15) 35.0 29.8
English, FLEURS (30) 4.9 4.9
Names in narrator 2's chapter spelled right 4 of 23 11 of 23

Words wrong for Whisper large-v3-turbo and Urdu Sukhan ASR v1 on every test and stress set

Examples

Public test clips transcribed by Whisper large-v3-turbo and by Urdu Sukhan ASR v1, run the way an application runs them (whisper.cpp, q8_0, beam 5, temperature fallback, no timestamps). Red marks a wrong word, [–] a missing one and [– N words] several, against the test set's reference after the same normalisation as our scores, so spelling and spacing count. ✅ marks the model with fewer mistakes on the clip and ❌ the one with more; a mistake is a wrong, extra or missing word. We picked these clips by hand to show where our model helps most, mostly from the top of a ranking of all clips by the number of words it fixed; the last example is one of the four clips where it did worse. Over all 47 clips with speech that we ran this way, our model was better on 27, worse on 4 and the same on 16 (19.1% against 37.8% words wrong).

1. Urdu then English (Common Voice + FLEURS) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference اسی میں ہے Due to the underwater topology the return flow is concentrated at a few deeper sections and a fast current to deep water may form there 28 words
Meaning “It is right in this”, then an English sentence. The base model wrote four English words for the whole clip; ours wrote both halves and got the last two words wrong.
Whisper large-v3-turbo You see me here. [– 24 words] ❌ 28 mistakes
Urdu Sukhan ASR v1 اسی میں ہے Due to the underwater topology the return flow is concentrated at a few deeper sections and a fast current to deep water may run away ✅ 2 mistakes

2. Urdu then English (Common Voice + FLEURS) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference احتساب عدالت کا فیصلہ کیا ہوتا ہے In just two weeks the Americans and Free French forces had liberated southern France and were turning towards Germany 26 words
Meaning “What is the accountability court's verdict?”, then an English sentence. The base model translated the Urdu into English, wrongly, instead of writing it down; ours wrote both halves without a mistake.
Whisper large-v3-turbo What is the decision of the government? In just two weeks, the Americans and free French forces had liberated southern France and were turning towards Germany. ❌ 7 mistakes
Urdu Sukhan ASR v1 احتساب عدالت کا فیصلہ کیا ہوتا ہے In just two weeks the Americans and free French forces had liberated Southern France and were turning towards Germany ✅ no mistakes

3. One- or two-word reply (IndicVoices) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference الگ الگ 2 words
Meaning “separately”. The base model wrote “Allah, Allah”.
Whisper large-v3-turbo اللہ اللہ ❌ 2 mistakes
Urdu Sukhan ASR v1 الگ الگ ✅ no mistakes

4. One- or two-word reply (IndicVoices) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference ہیلو 1 words
Meaning “hello”. The base model wrote “music”.
Whisper large-v3-turbo موسیقی ❌ 1 mistake
Urdu Sukhan ASR v1 ہیلو ✅ no mistakes

5. One- or two-word reply (IndicVoices) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference چھیاسی 1 words
Meaning “eighty-six”
Whisper large-v3-turbo چاہاں سی ❌ 2 mistakes
Urdu Sukhan ASR v1 چھیاسی ✅ no mistakes

6. IndicVoices, conversational · ✅ Urdu Sukhan ASR better

Transcript Result
Reference جی جی جی ہاں 4 words
Meaning “yes, yes, yes, yes”
Whisper large-v3-turbo دی دی دی ہاں ❌ 3 mistakes
Urdu Sukhan ASR v1 جی جی جی ہاں ✅ no mistakes

7. Quiet speech (a Common Voice clip, 30 dB softer) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference ٹام کار کے حادثے میں بچ گیا تھا 8 words
Meaning “Tom survived the car accident.” Both models wrote the name Tom as ٹوم.
Whisper large-v3-turbo ٹوم کار کی ہاتھ سے میں بچ کیا تھا ❌ 5 mistakes
Urdu Sukhan ASR v1 ٹوم کار کے حادثے میں بچ گیا تھا ✅ 1 mistake

8. Phone line (a Common Voice clip) · ✅ Urdu Sukhan ASR better

Transcript Result
Reference اس ماحول میں جو آزادی نظر آتی ہے 8 words
Meaning “the freedom that one sees in this atmosphere”
Whisper large-v3-turbo اس محول میں جو ازادی نظر آتی ہے ❌ 2 mistakes
Urdu Sukhan ASR v1 اس ماحول میں جو آزادی نظر آتی ہے ✅ no mistakes

9. UrduSpeech · one of the four clips where ours did worse · ❌ Urdu Sukhan ASR worse

Transcript Result
Reference گیا آنند وہار غریب رتھ ایکسپریس 6 words
Meaning “Gaya – Anand Vihar Garib Rath Express” (a train). Ours wrote “Anand Vihar” as one word.
Whisper large-v3-turbo گیا آنند ویہار غریب رتھ ایکسپریس ✅ 1 mistake
Urdu Sukhan ASR v1 گیا آنندوہار [–] غریب رتھ ایکسپریس ❌ 2 mistakes

No speech: on audio without speech, every Whisper decoder by default must write at least one word. Allowing an empty result (suppress_blank = false, or -nsb with our patch) only helps a model that has learned to stop. Any word here is a mistake.

Faint hiss and hum

default blank allowed
Whisper large-v3-turbo ❌ موسیقی (“music”) ❌ موسیقی
Urdu Sukhan ASR v1 ❌ اچھا (“all right”) ✅ (nothing)

Noise (MUSAN)

default blank allowed
Whisper large-v3-turbo ❌ موسیقی (“music”) ❌ موسیقی
Urdu Sukhan ASR v1 ❌ اچھا (“all right”) ✅ (nothing)

Clips: Common Voice (CC0), FLEURS (CC BY 4.0, Google), IndicVoices and UrduSpeech/Rasa (CC BY 4.0, AI4Bharat), MUSAN (CC BY 4.0); the quiet, phone and Urdu-then-English clips are Common Voice and FLEURS clips that we turned down, passed through a phone filter or joined, and the hiss is generated.

Recommended settings

Setting Why
Set the language (ur or en) when you know it whisper.cpp runs the encoder an extra time per 30 s window to detect the language, which doubles the time on short clips; detect once per file or conversation instead
Allow an empty result: suppress_blank = false (whisper.cpp), -nsb with our patch by default Whisper decoders forbid ending on the first token, so every Whisper model must write a word on silence; with this off, our model writes nothing on 100 noise, music and silence clips and no speech score changes
Voice detector (Silero VAD; whisper.cpp --vad -vsd 2000 -vp 300, faster-whisper vad_filter=True) skips non-speech; very faint speech can be dropped
No timestamps, no carried context (-nt -mc 0, condition_on_previous_text=False) avoids clips cut short at long pauses and errors carried from one window to the next
Beam search 5 with temperature fallback slightly better than greedy; fallback re-decodes a window that repeats itself
Names: list them in a prompt (--prompt "…" in whisper.cpp) raised the names spelled right in our hardest chapter from 12 to 17 of 23
On x86 servers: q4_0 and about 8 threads per job q4_0 was about 1.5× faster than q8_0 there; on a shared 12-core server, 12 threads were 3× slower than 8

No-speech clips on which whisper.cpp wrote at least one word, with default settings and with blank allowed

Speed

Compared with Whisper large-v3

Whisper large-v3 Urdu Sukhan ASR v1 (layout of large-v3-turbo)
Parameters 1,550 M 809 M
Encoder / decoder layers 32 / 32 32 / 4
whisper.cpp file, fp16 3,095 MB 1,625 MB
whisper.cpp file, q5_0 1,081 MB 574 MB
GPU memory (OpenAI's figure) about 10 GB about 6 GB
Relative speed (OpenAI's figure: English speech on an A100 GPU) 1× about 8×

File sizes are those in whisper.cpp's model repository; memory and speed are from OpenAI's Whisper README. The two models share the same 32-layer encoder; the speed-up comes from the decoder, which runs once for every output token. On a CPU in whisper.cpp the encoder does about 95% of the work (below), so the gain there is smaller. We have not timed or scored Whisper large-v3 ourselves.

Measured speed

The model has the layout of Whisper large-v3-turbo, so its speed is that model's. In whisper.cpp the encoder is about 95% of the work on a CPU, and it always processes a full 30-second window.

Machine (whisper.cpp v1.9.5, q8_0) Encoder, one 30 s window
Apple M4, GPU (Metal) 1.3 s
Apple M4, CPU, 4 threads 5.4 s
Cloud server, AMD EPYC, 12 vCPU 11.2 s
Intel Xeon E3 v2 (2012, no AVX2), 8 threads 21.1 s

Encoder time per 30-second window on four machines

On the cloud server, a 27-minute recording took 15.6 minutes with faster-whisper (int8, greedy, batches of 8; 1.76× real time) and 18.1 minutes with beam 5.

Model details

Developed by Asad Shehzad, Loudness.ai
Model type Encoder–decoder transformer (Whisper), 32 encoder and 4 decoder layers, 128 mel bins
Parameters 809 M
Languages Urdu, English, Urdu–English code-switching
Fine-tuned from openai/whisper-large-v3-turbo (MIT)
Licence CC BY-NC 4.0, non-commercial use; access approved on request; for commercial use, contact us. Whisper's MIT notice is in NOTICE.md
Version v1 (round 6 of training, step 2,775 of 3,697)

Uses

Intended: transcribing Urdu and mixed Urdu–English speech such as dictation, voice notes, interviews, lectures, audiobooks and short replies in voice interfaces; drafting transcripts for human proofreading.

Out of scope: speaker identification or diarisation; translation (untested); languages other than Urdu and English; transcripts used without human review where errors have serious consequences (medical, legal or financial records).

Training data

Full fine-tune on about 110 hours per pass (59,165 clips), sampled from 87,571 cleaned clips (about 151 hours):

Source Terms Clips per pass
IndicVoices Urdu (AI4Bharat) CC BY 4.0 20,000
Common Voice Urdu (Mozilla), via the Kaggle "Urdu dataset 20000" repack CC0 12,000
UAT audio only, transcribed by our round-4 model (confidence ≥ 0.85) no public licence; used by agreement with its developers 10,000
UrduSpeech (Urdu part of AI4Bharat's Rasa) CC BY 4.0 4,999
nzg73 Urdu_Audio-Realistic (Muhammad Noman) the dataset's own licence (speech models allowed, credit required, no misuse) 3,000
FLEURS Urdu train + dev; English train (Google) CC BY 4.0 2,370; 2,000
Two audiobooks read by two narrators; transcripts proofread by Asad Shehzad not redistributed 1,214 (521 clips ×2 or ×4)
Urdu–English clips joined from FLEURS and Common Voice as their sources 782
No-speech: MUSAN noise and instrumental music, generated hiss and hum CC BY 4.0 (some music tracks carry their own CC licences) 2,800

Clips were removed when their audio could not hold their text, when they were shorter than 0.5 s or longer than 30 s, or when Urdu text was mostly not in Urdu script. All text follows one Urdu house style (for example لیے rather than لئے) without punctuation. The audiobook transcripts come from a proofreading loop in which each model drafts and the author corrects.

Training procedure

Method full fine-tune of all weights
Steps 3,697 of 16 clips (one pass); released checkpoint: step 2,775, chosen with stress tests
Learning rate 5e-6, 10% linear warm-up, linear decay
Optimiser 8-bit AdamW (bitsandbytes), weight decay 0.01, gradient clipping 1.0
Precision fp32 master weights, bf16 computation, gradient checkpointing
Augmentation half of the clips: speed 0.9–1.1, gain, room echo, phone line, MUSAN noise and music; SpecAugment
Timestamps 30% of clips with known timings trained in timestamp form
Hardware one NVIDIA RTX A4000 (16 GB); 4.5 GPU-hours for this round

Evaluation

Seven held-out test sets (above) and nine stress sets: noise and instrumental music from MUSAN files held out from training, silence with hiss and hum, quiet speech, phone-line speech, one- or two-word replies, English, Urdu followed by English, and 25–30 seconds of continuous speech. Metrics: word and character error rate after normalisation; loops (runaway repetition) and invented text on no-speech clips (two or more words) are counted separately.

Our Common Voice, FLEURS and stress test sets are on Hugging Face as Loudness-ai/urdu-sukhan-asr-test-sets, with access on request, so you can check these numbers or score another model on the same clips.

Limitations

Read these before relying on the model.

  • Names are its weakest point. The test chapter of our second audiobook has 23 name occurrences: Urdu Sukhan ASR spelled 11 right, the base model 4. Names it rarely heard in training come out as similar common words, for example عمیر (Umair) as امیر (“rich”) and تانیا (Tania) as دانیا. A prompt listing the expected names raised the count from 12 to 17 of 23 in whisper.cpp. FLEURS news, full of foreign names, is where it improves least. When names matter (people, places, brands), give them in a prompt and check them by hand.
  • Word spacing and spelling: words are sometimes joined or split (آنندوہار for آنند وہار), and spelling follows our house style, which can differ from yours.
  • No punctuation: transcripts come without punctuation, like the training text.
  • Very faint English after Urdu can come out in Urdu letters or repeat a word.
  • Narrator-specific pronunciations (for example a dropped ھ: پھر heard as پر) are sometimes written as pronounced.
  • Silence: without the blank setting above, it writes a word such as اچھا on silence, as every Whisper model does.
  • Accents and settings: IndicVoices is mostly Indian Urdu, the Common Voice test has few women, and we have no test of spontaneous phone conversations between Pakistani speakers. Our test sets are small (20 to 300 clips) and we report no confidence intervals.
  • Speed on CPUs: every clip pays for a full 30-second window; short live clips are slower than real time on small CPUs.
  • Not measured: we have not scored or timed Whisper large-v3, measured memory use, or run the model on a phone.

Next version

What we plan to change for the next model, each from a problem this one showed:

Problem in v1 Plan for the next model
Names: 11 of 23 right in our test chapter; rare names become common words More name-rich training speech (news, interviews, stories with many characters), a larger test set of names, and name prompts in the examples and test page
Faint training speech taught the model to hear words in hiss Set every clip to a normal level before augmenting it
One word on silence unless an empty result is allowed Encode empty transcripts without a space, so the model can stop with default settings
Stress-test noise and music were heard in training Keep test noise and music out of augmentation
Joined or split words Check training transcripts for word spacing and count spacing errors separately
Every clip pays for a 30-second window on CPUs Fine-tune for shorter audio windows; train a smaller, faster student on more teacher-transcribed audio (our 16-layer student was 1.9× faster in the encoder but 0.5–5.4 points less accurate, so it is not released)
No iPhone build (the Core ML encoder lost precision) A precise on-device build for iPhone
Settings we have not tested Pakistani conversational and phone speech, more women speakers, larger test sets with confidence intervals, memory and phone measurements, and a direct comparison with Whisper large-v3

Environmental impact

The released round used 4.5 GPU-hours on one RTX A4000. All experiments, including five earlier rounds and a compression experiment that was not released, used about 23 A4000 GPU-hours and about 12 GPU-hours on an Apple M4 laptop. We did not measure energy use.

Citation 

@techreport{shehzad2026urdusukhan,
  author      = {Asad Shehzad},
  title       = {Urdu Sukhan {ASR}: Accurate {Urdu} Speech Recognition at Half the Size of {Whisper} large-v3},
  institution = {Loudness.ai},
  number      = {LA-RB-2026-004},
  year        = {2026},
  doi         = {10.5281/zenodo.23518703},
  url         = {https://loudness.ai/research/urdu-speech-recognition-whisper/}
}

Credits

Developed by Asad Shehzad, Loudness.ai. Audiobook transcripts proofread by Asad Shehzad. Base model: OpenAI Whisper large-v3-turbo (MIT). Data: AI4Bharat (IndicVoices, Rasa), Mozilla Common Voice, Google FLEURS, Muhammad Noman – nzg73, the UAT authors, and MUSAN (Snyder, Chen and Povey, 2015). Voice detection: Silero VAD.

Downloads last month
4
Safetensors
Model size
0.8B params
Tensor type
F16
·
Inference Providers NEW
Automatic Speech Recognition
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Loudness-ai/urdu-sukhan-asr-v1

Base model

openai/whisper-large-v3
Finetuned
openai/whisper-large-v3-turbo
Finetuned
(673)
this model

Datasets used to train Loudness-ai/urdu-sukhan-asr-v1

Evaluation results