SraVaani: How Vision Helps Hearing

How audio-image alignment — with no transcription labels — makes a multilingual Indian ASR model dramatically better.

Thousands of languages are spoken worldwide, yet most automatic speech recognition (ASR) systems cover only a handful. The bottleneck is almost always the same: you need large amounts of accurately transcribed speech, and for most languages, that data simply doesn't exist. Collecting it is expensive, slow, and requires skilled annotators who speak the language.

SraVaani is ARTPARK-IISc's answer to this problem for Indian languages. It's a multilingual model trained for ASR (STT) on 65 Indian languages and dialects, built on top of the Vaani dataset — one of the largest speech corpora for Indian languages. But the more interesting story isn't the scale. It's how we squeezed significantly better recognition out of the model without collecting a single additional transcription, by teaching it to align speech with images.

The Dataset: Vaani

156K speakers across 165 districts, talking about 288K images for 31K hours -- Vaani contains approximately 31,255 hours of speech across 105 Indian languages, of which 2,043 hours are transcribed. The data was collected using a picture-prompt paradigm: speakers were shown an image and asked to describe it, or simply to say whatever came to mind when looking at it.

This collection method was designed for accessibility — you don't need a scripted reading session or a professional studio — but it had an unexpected side effect: every utterance in the dataset comes with a semantically related image. The speaker looked at that image before speaking. The image and the audio describe overlapping content. That alignment, which was an artifact of the collection setup rather than anything we engineered, turned out to be the key to building a better model.

Self-Supervised Pretraining

Before any transcription data is introduced, the FastConformer encoder is pretrained from scratch on the full VAANI corpus: 29,912 hours of unlabelled speech spanning 105 languages. The objective is a wav2vec 2.0-style contrastive loss — for each masked frame, the model identifies its own target against 50 in-batch distractors. No transcriptions are used at this stage. Training runs for 70 epochs in FP32 with an effective batch size of 8,192, using AdamW with Noam annealing (peak LR ≈ 3.5×10⁻⁶). This gives the encoder a broad, language-agnostic representation of Indian speech before any task-specific fine-tuning.

The Architecture: FastConformer

The acoustic backbone of SraVaani is a 17-layer FastConformer encoder, a convolution-augmented transformer architecture designed for efficient and accurate speech recognition. FastConformer uses linearly scalable attention with reduced stride, giving it a strong capacity for modeling long-range acoustic dependencies while remaining practical to train and deploy.

For ASR fine-tuning, the encoder is paired with a hybrid CTC-TDT decoder — a combination that supports both accurate beam search decoding (via TDT) and fast greedy decoding (via CTC). TDT durations of {0, 1, 2, 3, 4} are used, with CTC contributing 30% of the training loss. The tokenizer uses BPE with a vocabulary of 5,000 tokens derived from the transcript corpus of the train split of Vaani.

The Key Innovation: Audio-Image Alignment

The conventional ASR training pipeline has two stages: SSL pretraining on raw audio, then supervised fine-tuning on transcribed data. We introduce a third stage between the two:

The idea is to use the Vaani image-audio pairs as a transcription-free adaptation signal. Before the model ever sees a transcript, we take the pretrained encoder and train it to align its audio representations with semantic representations extracted from a frozen vision encoder. Audio and image of the same Vaani prompt should produce similar embeddings; unrelated pairs should not.

This is trained with a sigmoid SigLIP contrastive loss using in-batch negatives. The weights from this alignment stage are then carried forward as the initialization for supervised ASR fine-tuning.

The intuition: images carry rich semantic information about the content of speech. If the audio encoder learns to produce representations that are compatible with image semantics, it's building a richer, more structured representation of what's being said — before it ever sees a transcript to tell it what words were spoken. That richer representation then gives the ASR decoder a better foundation to work from.

Critically, the 11.8M audio-image pairs used for alignment (16,580 hours) are drawn from the same Vaani pretraining corpus — no new audio is introduced. The gains come purely from the alignment stage, not from additional data exposure.

The Alignment Setup

For the alignment stage, we use SigLIP2-large (patch16-384, ~650M parameters) as the frozen vision encoder. The audio encoder produces 16 query vectors via multi-query attention pooling, projected by an MLP to a 1024-dimensional shared space. The image encoder produces up to 16 patch token representations. The pair score is computed with an asymmetric MaxSim: for each audio query, take the maximum cosine similarity over the image tokens, then average across all 16 audio queries. The whole system is trained with a sigmoid SigLIP contrastive loss using in-batch negatives.

All 17 FastConformer encoder blocks are kept trainable during alignment, while the vision encoder is frozen throughout. Training uses AdamW with a batch size of 64 across 8×H100 GPUs for 200,000 steps.

Supervised Fine-Tuning

The third and final stage takes the alignment-stage encoder and trains the full ASR model on 30,565 hours of transcribed multilingual Indian speech from 18 public datasets spanning 65 languages and dialects, with 698 hours held out for validation. A freshly initialized hybrid CTC-TDT decoder is attached to the aligned encoder, and all weights are updated end-to-end.

The 65 languages, grouped by training data volume:

Key training configuration:

Audio inputs are 128-dimensional log-mel spectrograms (25ms window, 10ms stride) with subsampling factor 8, so the encoder sees one frame every 80ms. SpecAugment is applied during training for regularization. The best checkpoint by validation WER is selected as the released model.

Results

Evaluating on a held-out Vaani test partition covering 48 languages, the audio-image alignment stage consistently outperforms the unaligned baseline: WER drops from 28.09% to 27.40% (a 2.47% relative improvement, statistically significant at p < 10⁻⁴ by paired bootstrap). SigLIP-large improves 37 out of 48 languages, with 20 of those improvements statistically significant and zero statistically significant regressions.

The figure below shows WER across all 49 evaluated Indic languages, comparing SraVaani-1.0 against Gemini 3 Flash, Sarvam Saaras v3, and IndicConformer-600M-Multilingual. Languages are ordered left to right by the cross-system mean WER; shaded columns are languages that appear in the Vaani benchmark only (no other public benchmark exists for them). A missing marker means the system produced no output for that language.

SraVaani matches or beats both Gemini 3 Flash and Sarvam Saaras v3 on the large majority of languages. The gains are most pronounced on lower-resource and harder languages — Maithili, Nepali, Tamil, Malayalam — and SraVaani is the only system that produces output for the 51 tribal and dialect languages in the shaded region, where the other three systems either produce no output or exceed 80% WER. Against IndicConformer — a strong multilingual RNN-T baseline — SraVaani is competitive across the board, with a mean WER of 28.4% vs 30.2% over the 17 languages where direct comparison is possible.

Try It

SraVaani-1.0 is available on Hugging Face:

ARTPARK-IISc/SraVaani-1.0

The model is a hybrid CTC-TDT FastConformer checkpoint in NeMo format. To get started with fine-tuning it on your own data, see the sravaani repository — it includes a step-by-step notebook and scripts for running training, and evaluating against a baseline.

Citation

If you use SraVaani or build on this work, please cite:

@misc{pulikodan2026vaanicapturinglanguagelandscape,
      title={VAANI: Capturing the language landscape for an inclusive digital India}, 
      author={Sujith Pulikodan and Abhayjeet Singh and Agneedh Basu and Nihar Desai and Pavan Kumar J and Pranav D Bhat and Raghu Dharmaraju and Ritika Gupta and Sathvik Udupa and Saurabh Kumar and Sumit Sharma and Vaibhav Vishwakarma and Visruth Sanka and Dinesh Tewari and Harsh Dhand and Amrita Kamat and Sukhwinder Singh and Shikhar Vashishth and Partha Talukdar and Raj Acharya and Prasanta Kumar Ghosh},
      year={2026},
      eprint={2603.28714},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2603.28714}, 
}

SraVaani is developed at SPIRE LAB & ARTPARK (AI & Robotics Technology Park), I-Hub @ IISc, Bangalore.