tidelganesh/svara-tts-v1-bnb-8bit

bitsandbytes int8 weights re-quantization of kenpath/svara-tts-v1. Linear-layer weights stored in int8 with vector-wise quantization, computed in bf16. No calibration dataset; no fine-tuning. Quality is near-lossless relative to the upstream model — listen-test a few reference prompts before production.

Why int8

	bf16	int8 (this repo)
VRAM	~16 GB	~8–10 GB
Throughput vs bf16	1.0×	~0.7–1.0× (dequant-bound on Ampere/Ada)
Quality	baseline	near-lossless, occasional edge-case artifacts
Calibration data	n/a	not required (post-training weight-only quant)

int8 is the right choice when you want more concurrent streams per GPU or you're deploying to a smaller card. On a 24 GB GPU (L4, RTX 4090) int8 leaves room for long-context KV cache and 3–4 concurrent requests. On a 16 GB card (T4) int8 is required, bf16 will OOM.

Hardware

GPU	int8 fit	Notes
A100 40/80 GB	✅ very comfortable	bf16 likely better here — VRAM is free
A40 48 GB	✅ very comfortable	bf16 likely better — VRAM is free
L4 24 GB	✅ tight	best use case — fits with KV cache headroom
RTX 4090 24 GB	✅ tight	best use case
T4 16 GB	✅	required to fit at all
RTX 3090 24 GB	✅	works

Software: CUDA ≥ 11.8, PyTorch ≥ 2.3, transformers >= 4.45, accelerate, bitsandbytes >= 0.43, flash-attn (recommended).

Usage

import torch
from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModel.from_pretrained(
    "tidelganesh/svara-tts-v1-bnb-8bit",
    trust_remote_code=True,
    quantization_config=bnb,
    device_map="cuda",
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(
    "tidelganesh/svara-tts-v1-bnb-8bit",
    trust_remote_code=True,
)

trust_remote_code=True is required because the upstream model ships custom modeling code in modeling_svara.py. The quantization_config is required — from_pretrained will not auto-detect int8 weights.

Serving

This repo is intended to back a self-hosted Svara-TTS FastAPI server. Point the server's model loader at this repo instead of the upstream one:

- model = AutoModel.from_pretrained("kenpath/svara-tts-v1", trust_remote_code=True)
+ from transformers import BitsAndBytesConfig
+ model = AutoModel.from_pretrained(
+     "tidelganesh/svara-tts-v1-bnb-8bit",
+     trust_remote_code=True,
+     quantization_config=BitsAndBytesConfig(load_in_8bit=True),
+     device_map="cuda",
+     attn_implementation="flash_attention_2",
+ )

Server contract (POST /v1/text-to-speech, 24 kHz PCM16 mono) is unchanged.

How it was made

pip install -U "transformers>=4.45" accelerate bitsandbytes "huggingface_hub[cli]" hf_transfer
pip install -U flash-attn --no-build-isolation
export HF_HUB_ENABLE_HF_TRANSFER=1
huggingface-cli login
export HF_TOKEN=hf_xxx...

python backend/quantize_and_push_svara_8bit.py   # ~15–25 min on A40
python backend/verify_svara_8bit.py              # round-trip check

The script:

1. Loads kenpath/svara-tts-v1 with BitsAndBytesConfig(load_in_8bit=True) and Flash-Attention 2.
2. Saves with safe_serialization=True (safetensors) sharded at 2 GB.
3. Copies the upstream tokenizer / preprocessor / *.py files so trust_remote_code still works.
4. Pushes everything via model.push_to_hub + tokenizer.push_to_hub to this repo.

Quality notes

• bitsandbytes int8 (LLM.int8()) has a known numerical edge case on Llama-3 hidden dims of 4096 with vector_dim=4096 — Svara's Llama-3.1-8B sits right on this boundary. Outlier features in the LM head can occasionally cause audible artifacts on long or rare-token Tamil/English inputs.
• Mitigation: switch to bf16 (tidelganesh/svara-tts-v1-bf16) if you observe:
  • skipped/repeated syllables
  • garbled phonemes on English code-mixed inputs
  • silence mid-sentence on long generations
• The audio decoder (VITS-style) is not quantized — only the LM is int8. The decoder is fp32 in both bf16 and int8 repos.

Limitations

• int8 is weight-only quantization. Activations stay in bf16; KV cache stays in bf16. Do not expect int8-level memory savings on the KV cache.
• Throughput on Ampere/Ada is roughly bf16-parity — int8 wins on VRAM, not on tokens/sec.
• Custom modeling code is re-shipped from the upstream repo verbatim. If kenpath/svara-tts-v1 updates its modeling_svara.py, this repo must be re-pushed.

Provenance

• Base model: kenpath/svara-tts-v1
• Conversion: bitsandbytes load_in_8bit=True + safetensors re-serialization
• Tooling: transformers, accelerate, bitsandbytes, huggingface_hub, flash-attn

Related

• tidelganesh/svara-tts-v1-bf16 — bf16 cast (lossless, ~16 GB VRAM). Recommended default when VRAM is not the bottleneck.
• kenpath/svara-tts-v1 — upstream fp32/fp16 checkpoint.

License

Inherits the upstream license. Please refer to kenpath/svara-tts-v1 for terms.