Fine-Tune SraVaani on Your Own Speech Data

A step-by-step guide to adapting a 65-language FastConformer speech recognizer to a new language or domain, on a single 15GB GPU.

---

This is a step-by-step guide to fine-tuning SraVaani — a Hybrid RNN-T + CTC FastConformer model pretrained on 65 Indian languages by ARTPARK-IISc — on your own speech data. The pipeline is language-agnostic: bring your own audio and transcripts, follow the steps, and you'll have a fine-tuned checkpoint at the end.

To make the walkthrough concrete, we use Wancho as a running example — a Tibeto-Burman language spoken in Arunachal Pradesh with around 50,000 speakers and limited digital resources — but every step applies equally to any language or domain you want to adapt the model to.

What you'll learn:

• How to load and fine-tune a NeMo Hybrid RNN-T + CTC model on new data
• How to prepare data in NeMo's tarred format
• How to compare a baseline and fine-tuned model on the same test set

What you'll need:

• Base model: SraVaani-1.0 — Hybrid RNN-T + CTC BPE FastConformer, 63 Indian languages
• Your own dataset (or ARTPARK-IISc/Vaani-transcription-part to follow along with the Wancho example)
• 1× NVIDIA GPU with at least 15GB VRAM (we used a T4); ~5GB free disk (checkpoint ~1.8GB, Wancho data ~1.4GB)
• A Hugging Face account and access token if using the Vaani dataset
• NVIDIA NeMo + PyTorch Lightning

The full notebook is available on GitHub.

---

1. Environment Setup

Create a virtual environment and install NeMo with ASR support:

python3 -m venv venv
source venv/bin/activate
pip install "nemo_toolkit[asr,cu13]" pytorch-lightning omegaconf

Gotcha: always use the cu12/cu13 extra. pip install nemo_toolkit[asr] alone is not enough. NeMo's RNN-T/TDT loss needs the numba-cuda GPU kernel backend, which only comes via the cu12/cu13 extra (match your CUDA driver version from nvidia-smi). Without it, everything works until the first training step, which then fails deep in a numba kernel compile with an unhelpful TypeError: Signature mismatch. Fix: pip install "nemo_toolkit[asr,cu13]".

Verify your install:

import torch
print("torch:", torch.__version__, "| CUDA available:", torch.cuda.is_available())

import nemo
print("nemo_toolkit:", nemo.__version__)

# If this import fails, you hit the cu12/cu13 gotcha above.
import numba_cuda
print("numba-cuda: OK")

---

2. Download the Pretrained Checkpoint

The .nemo checkpoint (~1.8GB) is hosted on Google Drive. Use gdown to handle the large-file confirmation flow that a plain wget/curl chokes on:

pip install gdown
gdown "1v5VaYibAaDSFuWvROsPbCzeG3iM6VbxY" -O SraVaani-nemo-checkpoint.nemo

Alternatively, download it manually from Google Drive and place it in your project root. Don't have access? Request access here.

Always verify the download. A .nemo file is a tar archive; a truncated download still produces a file that looks plausible by size but fails the moment NeMo tries to read past the truncation point:

import tarfile

with tarfile.open("SraVaani-nemo-checkpoint.nemo") as tf:
    print(tf.getnames())  # raises tarfile.ReadError on a truncated file

If this raises ReadError: unexpected end of data, re-download. Don't debug the training script first — we lost time doing exactly that before checking the checkpoint itself.

---

3. Prepare the Dataset

NeMo expects audio at 16kHz mono 16-bit PCM, paired with a JSONL manifest — one JSON line per clip. There are two ways to set this up depending on your situation.

Option A: Plain files in a folder (simpler)

If your audio is already on disk, just point the manifest at the actual file paths:

{"audio_filepath": "/data/train/clip_0000.wav", "text": "pha ama gaga aga gih pan tega mai tai", "duration": 5.547}

Here audio_filepath is a regular filesystem path. In the data loader config (§6) you leave is_tarred as False and don't set tarred_audio_filepaths. This is the easiest starting point if you're experimenting or your dataset fits comfortably on disk.

Option B: Tarred shards (recommended for larger datasets)

NeMo's tarred format pairs a tarred audio archive (shard_NNNN.tar) with a matching manifest (shard_NNNN.json). The manifest entries look the same, except audio_filepath is the filename inside the tar rather than a filesystem path:

{"audio_filepath": "clip_0000.wav", "text": "pha ama gaga aga gih pan tega mai tai", "duration": 5.547}

Tarred shards improve throughput at scale by replacing many small file reads with sequential reads from a few large archives — relevant once your dataset is several hours of audio or you're training across multiple GPUs. For a first fine-tuning run on a few hours of data, Option A is fine.

In the Wancho example, we pull data from ARTPARK-IISc/Vaani-transcription-part and pack it into shards. How you get there depends on where your data lives — the key requirement is 16kHz mono 16-bit PCM audio with a matching JSONL manifest.

Directory layout for Option B:

data/
  train/
    shard_0000.tar
    shard_0000.json
  validation/
    shard_0000.tar
    shard_0000.json
  test/
    shard_0000.tar
    shard_0000.json

---

4. Load the Pretrained Model

from nemo.collections.asr.models import EncDecHybridRNNTCTCBPEModel

NEMO_CHECKPOINT = "SraVaani-nemo-checkpoint.nemo"

model = EncDecHybridRNNTCTCBPEModel.restore_from(NEMO_CHECKPOINT)

Once the model is loaded, you have two fine-tuning strategies:

Option 1 — Freeze the encoder (decoder/joint only): Call model.encoder.freeze() after loading. This trains only the decoder and joint network — the parts that are most language-specific — while leaving the acoustic encoder unchanged. It's the safer choice when your dataset is small, since a frozen encoder can't overwrite the general-purpose phonetic representations the model built during pretraining.

Option 2 — Full fine-tuning (encoder + decoder): Skip the freeze call entirely. All weights are updated, which can close the gap if your target language's phonetics are genuinely different from the pretraining mix — but it requires more GPU memory and a lower learning rate to avoid destabilizing the encoder.

Which should you choose? If your dataset is small (a few hours) or your target language is phonetically close to the pretraining mix, freeze the encoder — the risk of catastrophic forgetting outweighs the potential gains from end-to-end updates. In the Wancho example (~5–9 hours of audio), we freeze the encoder for exactly this reason.

model.encoder.freeze()   # omit this line for full fine-tuning
print("Encoder frozen — only decoder/joint weights will be updated.")

Choosing the Training Decoder

EncDecHybridRNNTCTCBPEModel has both an RNN-T and a CTC decoder head. During training, ctc_weight in the model config controls which losses are active:

with open_dict(model.cfg):
    model.cfg.model_defaults.ctc_weight = 0.3   # hybrid: loss = 0.7×RNNT + 0.3×CTC (default)
    # model.cfg.model_defaults.ctc_weight = 0.0 # pure RNN-T
    # model.cfg.model_defaults.ctc_weight = 1.0 # pure CTC

For inference, you can independently choose which decoder head to use for decoding:

model.change_decoding_strategy(decoder_type="rnnt")  # RNN-T beam/greedy search
model.change_decoding_strategy(decoder_type="ctc")   # CTC greedy/beam search (faster)

The key benefit of hybrid training (ctc_weight=0.3) is that it trains both heads simultaneously, so at inference time you can choose either: RNN-T for best accuracy, or CTC for faster greedy decoding. If you set ctc_weight=0.0 or 1.0, only one head is trained and only that head should be used at inference.

---

5. Configure Data Loaders and Optimizer

Several choices here are tuned specifically for a small, single GPU rather than for throughput:

from omegaconf import OmegaConf, open_dict

# Option B (tarred shards) — use these if you followed §4 Option B
TRAIN_SHARDS    = "data/train/shard_0000.tar"
TRAIN_MANIFESTS = "data/train/shard_0000.json"
VAL_SHARDS      = "data/validation/shard_0000.tar"
VAL_MANIFESTS   = "data/validation/shard_0000.json"

# Option A (plain files) — if you used §4 Option A, set these instead and
# remove the tarred_audio_filepaths lines and set is_tarred = False below.
# TRAIN_MANIFESTS = "data/train/manifest.json"
# VAL_MANIFESTS   = "data/validation/manifest.json"

MAX_EPOCHS = 2
BATCH_SIZE = 4      # kept small to fit in T4 VRAM
NUM_WORKERS = 1     # kept small to avoid CPU memory pressure from audio workers
ACCUMULATE = 8      # effective batch size = BATCH_SIZE × ACCUMULATE = 32

with open(TRAIN_MANIFESTS, encoding="utf-8") as f:
    n_train = sum(1 for line in f if line.strip())

# ceil(n_train / BATCH_SIZE) → batches/epoch → // ACCUMULATE → opt steps/epoch → × epochs
MAX_STEPS = (-(-n_train // BATCH_SIZE) // ACCUMULATE) * MAX_EPOCHS
print(f"{n_train} train clips → {MAX_STEPS} total optimizer steps over {MAX_EPOCHS} epochs")

with open_dict(model.cfg):
    model.cfg.train_ds.manifest_filepath    = TRAIN_MANIFESTS
    model.cfg.train_ds.tarred_audio_filepaths = TRAIN_SHARDS
    model.cfg.train_ds.is_tarred            = True
    model.cfg.train_ds.shuffle_n            = 2048
    model.cfg.train_ds.batch_size           = BATCH_SIZE
    model.cfg.train_ds.num_workers          = NUM_WORKERS
    model.cfg.train_ds.pin_memory           = True

    model.cfg.validation_ds.manifest_filepath      = VAL_MANIFESTS
    model.cfg.validation_ds.tarred_audio_filepaths = VAL_SHARDS
    model.cfg.validation_ds.is_tarred              = True
    model.cfg.validation_ds.batch_size             = BATCH_SIZE
    model.cfg.validation_ds.num_workers            = NUM_WORKERS

    # Low LR: we're nudging a pretrained model, not training from scratch
    model.cfg.optim.lr           = 1e-4
    model.cfg.optim.name         = "adamw"
    model.cfg.optim.weight_decay = 1e-3
    model.cfg.optim.sched        = OmegaConf.create({
        "name":         "CosineAnnealing",
        "warmup_steps": 10,
        "max_steps":    MAX_STEPS,
        "min_lr":       1e-6,
    })

model.setup_training_data(model.cfg.train_ds)
model.setup_validation_data(model.cfg.validation_ds)
model.setup_optimization(model.cfg.optim)

A few notes on the choices:

• BATCH_SIZE = 4 + ACCUMULATE = 8 gives an effective batch size of 32 while keeping GPU memory usage tractable on a T4. RNN-T has high memory overhead because it computes a loss over a 3D (batch × time × label-length) lattice; small per-step batch sizes are normal here.
• NUM_WORKERS = 1 avoids the CPU memory pressure that comes from audio workers each holding a copy of the decoded audio buffer. With tarred data and short clips, a single worker is rarely the throughput bottleneck.
• lr = 1e-4 with warmup keeps the learning rate conservative. You're nudging parameters that are already well-converged on a related task, not training from a random initialization.

---

6. Train

A MetricsLogger callback prints train_loss every step and val_wer at every validation pass — useful when you're watching a run live rather than reviewing TensorBoard after the fact. exp_manager handles checkpointing (top-3 by val_wer) and TensorBoard logging.

import lightning.pytorch as pl
from nemo.utils.exp_manager import exp_manager


class MetricsLogger(pl.Callback):
    """Prints train_loss every step and val_wer at every validation pass."""

    def on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx):
        loss = trainer.callback_metrics.get("train_loss")
        if loss is not None:
            print(f"[step {trainer.global_step}] train_loss={loss.item():.4f}")

    def on_validation_epoch_end(self, trainer, pl_module):
        wer = trainer.callback_metrics.get("val_wer")
        if wer is not None:
            print(f"[step {trainer.global_step}] val_wer={wer.item():.4f}")


trainer = pl.Trainer(
    devices=1,
    accelerator="gpu",
    max_epochs=MAX_EPOCHS,
    accumulate_grad_batches=ACCUMULATE,
    gradient_clip_val=1.0,
    val_check_interval=1.0,
    log_every_n_steps=10,
    logger=False,
    enable_checkpointing=False,
    callbacks=[MetricsLogger()],
)

exp_manager(trainer, {
    "exp_dir": "experiments/",
    "name": "sra-vaani-finetune",
    "create_tensorboard_logger": True,
    "create_checkpoint_callback": True,
    "checkpoint_callback_params": {
        "monitor": "val_wer",
        "mode": "min",
        "save_top_k": 3,
    },
})

trainer.fit(model)
model.save_to("sra-vaani-finetuned.nemo")

---

7. Monitor Training

Everything logged during trainer.fit() also lands in the TensorBoard event file under experiments/. You can pull it directly with EventAccumulator for a quick static plot — useful when you don't want to launch a separate TensorBoard server:

from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
import glob
import matplotlib.pyplot as plt

event_file = glob.glob("experiments/sra-vaani-finetune/*/events.out.tfevents.*")[-1]
ea = EventAccumulator(event_file, size_guidance={"scalars": 0})
ea.Reload()

train_loss = ea.Scalars("train_loss")
steps  = [e.step  for e in train_loss]
values = [e.value for e in train_loss]

fig, ax = plt.subplots(figsize=(9, 5))
ax.plot(steps, values, linewidth=2)
ax.set_xlabel("Optimizer step")
ax.set_ylabel("train_loss")
ax.set_title("SraVaani fine-tune — training loss (Wancho)")
ax.grid(True, axis="y", alpha=0.3)
plt.tight_layout()
plt.show()

---

8. Evaluate: Original vs. Fine-Tuned

The real test is comparing the original and fine-tuned checkpoints on the same held-out test set, with the same text normalization applied to both references and hypotheses before scoring. Skipping the normalization step means you're partly measuring formatting differences, not transcription quality.

import csv, json, re, shutil, tarfile, tempfile
import jiwer
from nemo.collections.asr.models import EncDecHybridRNNTCTCBPEModel


def clean(text):
    """Lowercase and collapse whitespace. Adapt this to your language's needs or your data specifics."""
    text = text.lower().strip()
    text = re.sub(r"\s+", " ", text)
    return text

TEST_SHARDS         = "data/test/shard_0000.tar"
TEST_MANIFEST       = "data/test/shard_0000.json"
ORIGINAL_CHECKPOINT = "SraVaani-nemo-checkpoint.nemo"
FINETUNED_CHECKPOINT = "sra-vaani-finetuned.nemo"
INFER_BATCH_SIZE    = 32


def load_test_set(shards_path, manifest_path):
    tmp_dir = tempfile.mkdtemp(prefix="wancho_test_")
    with tarfile.open(shards_path) as tar:
        tar.extractall(tmp_dir)
    audio_filepaths, references = [], []
    with open(manifest_path, encoding="utf-8") as f:
        for line in f:
            if line.strip():
                entry = json.loads(line)
                audio_filepaths.append(f"{tmp_dir}/{entry['audio_filepath']}")
                references.append(entry["text"])
    return audio_filepaths, references, tmp_dir


def run_inference(checkpoint_path, audio_filepaths):
    m = EncDecHybridRNNTCTCBPEModel.restore_from(checkpoint_path)
    m.eval()
    if torch.cuda.is_available():
        m = m.cuda()
    hyps = m.transcribe(audio=audio_filepaths, batch_size=INFER_BATCH_SIZE)
    hyps = [h.text if hasattr(h, "text") else h for h in hyps]
    del m
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
    return hyps


audio_filepaths, references, tmp_dir = load_test_set(TEST_SHARDS, TEST_MANIFEST)
print(f"Loaded {len(audio_filepaths)} test clips")

hyp_original  = run_inference(ORIGINAL_CHECKPOINT,  audio_filepaths)
hyp_finetuned = run_inference(FINETUNED_CHECKPOINT, audio_filepaths)
shutil.rmtree(tmp_dir, ignore_errors=True)

# Apply the same clean() normalization to refs and both hypothesis sets
refs_clean          = [clean(r) for r in references]
hyp_original_clean  = [clean(h) for h in hyp_original]
hyp_finetuned_clean = [clean(h) for h in hyp_finetuned]

wer_original  = jiwer.wer(refs_clean, hyp_original_clean)
wer_finetuned = jiwer.wer(refs_clean, hyp_finetuned_clean)

print(f"Original checkpoint   WER: {wer_original:.4f}  ({wer_original  * 100:.2f}%)")
print(f"Fine-tuned checkpoint WER: {wer_finetuned:.4f} ({wer_finetuned * 100:.2f}%)")

# Save a per-clip breakdown for offline analysis
with open("inference_results.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["audio_filepath", "reference", "hyp_original", "hyp_finetuned"])
    for path, ref, ho, hf in zip(audio_filepaths, refs_clean,
                                  hyp_original_clean, hyp_finetuned_clean):
        writer.writerow([path.rsplit("/", 1)[-1], ref, ho, hf])

---

9. What to Expect

Once your run completes, compare baseline and fine-tuned WER on your held-out test set. On the Wancho example:

A 1-point absolute improvement from a short decoder-only fine-tune on a few hours of audio is the expected order of magnitude. The encoder (unchanged) is still doing the acoustic heavy lifting on a language it saw little of during pretraining — 2 epochs is enough to adapt the output vocabulary mapping but not enough to reshape the prediction network's internal language model.

Don't be discouraged by a modest first result. The pipeline being validated is the win: the model loads, the data flows, training converges, the fine-tuned checkpoint outperforms the baseline. Everything after this is scale and experimentation — see §11 for what to try next.

---

10. What Would Actually Move the Needle

• Unfreeze the encoder. The frozen-encoder result establishes a baseline and confirms the setup. Full fine-tuning — lower LR, more memory, more epochs — is the natural next step once you know the decoder isn't your bottleneck.
• Train longer on more data. Two epochs on a few hours of audio is a smoke test. The loss curve plateauing quickly is expected at this scale; with more data or more epochs, there's room to run.
• Repeat for other underrepresented languages. The pipeline here (download → clean → tar/manifest → freeze-and-finetune → compare-to-baseline) is language-agnostic. Only the dataset config and the checkpoint path need to change for any other language in the Vaani dataset.
• Try it on your own data. Everything here works for any language or domain once you've prepared your data in NeMo's tarred format (§3). Swap in your own audio and transcripts and the rest of the notebook runs unchanged.

---

Conclusion

SraVaani is a capable foundation model for Indian-language ASR, and NeMo makes it straightforward to adapt to a new language or domain. The key friction points — the cu12/cu13 install gotcha, the tarred data format, the memory budget on a small GPU — are all solvable with the notes above.

Swap in your own audio and transcripts, follow the same steps, and the rest of the pipeline runs unchanged. If you build on this for another language or domain, we'd love to hear how it goes. Questions? Reach out at [email protected] or open an issue on the GitHub repo.

---

SraVaani is developed at SPIRE LAB & ARTPARK (AI & Robotics Technology Park), I-Hub @ IISc, Bangalore.