loom-ai-org/vocetta2-1m-loom

Vocetta2 1M

VantoraLabs' Vocetta2 1M: a complete English text-to-speech model in 1,031,754 parameters -- a duration net, an acoustic net to a mel spectrogram and a ConvNeXt/iSTFT decoder at 24 kHz -- exported for loom.cpp.

This is a loom.cpp export: a single self-describing GGUF that carries its own graph topologies, tokenizer (if any) and driver script, produced by loom-exporter.

Original model

Exported from VantoraLabs/Vocetta2-1m. The F32 file carries the same parameters unmodified, in loom.cpp's GGUF format; the others are quantized from them (see Files).

License

mit, inherited from the base model above.

Language(s)

en

Usage

Run it with loom-py -- loom-py-rt on PyPI:

pip install -U "loom-py-rt[hub,phonemes]"

Needs loom-py-rt 1.0.0rc18 or later. This file uses the engine's ISTFT op, new in that release; an older runtime stops with unknown op 'ISTFT'.

NOTE: This is a work in progress. For now, in order to avoid license conflicts and keep dependencies at a minimum, we opted for orthography2ipa as our "swiss-knife" phonemizer. For deep-orthography languages like English, to get stressing rules and context-based phonemization, the phonemizer must register a reference lexicon (e.g., ipa-dict) to get highly accurate phonemization. Full quality can be achieved by phonemizing the text yourself using your engine of choice and feeding it to the model via the argument phonemes as in the example below.

import loom

model = loom.Model.from_pretrained("loom-ai-org/vocetta2-1m-loom", "vocetta2-1m-f32.gguf")

# The built-in text door: orthography2ipa, plus the US English pronunciation lexicon that ships in this
# repo (hexgrad/misaki's, Apache-2.0). The model declares the IPA conventions it was trained in and the
# door rewrites the G2P's output to them before encoding.
loom.phonemizers.set_lexicon("https://huggingface.co/loom-ai-org/vocetta2-1m-loom/resolve/main/lexicons/en_US-misaki.tsv", language="en-US")
audio = model.text2speech.infer("hello world")
audio.save("hello.wav")

# Full quality: misaki-style IPA from your own G2P is encoded as given.
# audio = model.text2speech.infer(phonemes="...")

The layer underneath

The call above is the high-level door: one per task, named for the modality pair it maps between, with the windowing, sampling and assembly this model needs already applied. Under it, model.infer(...) passes your arguments straight to the driver this GGUF embeds -- which is where you go for a knob the door does not name.

model.driver_source prints that driver, including a header comment documenting every argument it accepts for this model, and is the authority on it. See loom-py for the API and loom.cpp for what the engine does between the two.

Known limitations

The text door is not upstream's front end. Through text2speech.infer(text) with the bundled lexicon, this model scored 9.8% word error rate on 30 LibriSpeech sentences (Whisper small) against 7.2% with upstream's own G2P (dictionaries plus a neural fallback for unknown words). Without the lexicon, English is far worse. For upstream's quality, phonemize with upstream's microtts.g2p and pass phonemes=.

At most 207 phoneme symbols per call (upstream's own limit, BOS and EOS included); split longer text at its sentences. English only, one voice.

Upstream's 2400-frame cut is not reproduced. Upstream truncates the spectrogram at 2400 frames (25.6 s); this export synthesises every frame the durations ask for. Below 2400 frames, which upstream's own durations essentially always are, the audio is upstream's.

The decoder is fed zero noise by default, as upstream renders it. noise_scale= (with seed=) adds Gaussian noise, which upstream measured as worse on every metric; length_scale= slows or speeds the speech.

Files

One model in several precisions; every file carries the same graph and driver. The quantized ones pack the convolution and projection weights (loom-export --quantize <type>), so their numbers differ from the original model's by the amount noted.

• vocetta2-1m-f32.gguf -- full precision, 4.2 MB; upstream's waveform to 1.8e-4, upstream's own f32-vs-f64 spread.
• vocetta2-1m-q8_0_pc.gguf -- 2.9 MB: the decoder's pointwise layers and head in per-channel int8; the acoustic net stays F32, since its width-72 convolutions would need a third more zeros to quantize. Whisper's word error rate on 60 syntheses is 8.5% against F32's 8.5%. On a Pi Zero it runs at F32's speed and on a Pi 4 slower, and its peak memory is 1-4 MB below F32's 57-58 MB, nearly all of which is the runtime: a smaller download rather than a faster or leaner model.
• lexicons/en_US-misaki.tsv -- hexgrad/misaki's US English pronunciation dictionaries (us_gold.json over us_silver.json, each word's default reading, the same dictionaries upstream's front end uses), Apache-2.0 (lexicons/LICENSE.misaki), as word<TAB>ipa -- the shape loom.phonemizers.set_lexicon reads.