NbAiLab/nb-asr-qwen17b-bokfink021-beta

NB-ASR Bokfink 0.2.1-beta — Qwen3-ASR 1.7B

This repository is version 0.2.1-beta of the Qwen3-ASR 1.7B model in the NB-ASR project at the National Library of Norway. It provides reading-optimised and verbatim Norwegian transcription, Bokmål/Nynorsk output, and English translation of Norwegian speech. It also trains on long and sparse speech, contextual vocabulary, and intentionally empty transcripts.

This is a public, gated beta, included in the NB-ASR beta collection. Request access through the collection; its shared access policy applies to both model sizes. Model access does not grant access to the underlying speech data.

Both 0.2.1-beta models are the final 26,188-step / two-epoch checkpoints, trained independently from their matching custom-tokenizer bases on the same corpus. They use augmentation probability 0.50 and no BPE dropout. The separate BPE-dropout experiment is not part of this release. The companion model is NbAiLab/nb-asr-qwen06b-bokfink021-beta. In every comparison below, bold identifies this repository, not necessarily the best score. Family names 0.6B/1.7B follow the upstream Qwen names; the complete stored weights here contain 2,038,052,480 parameters, including the audio components.

What to use it for

Input	Requested output	Control
Norwegian speech	Reading-optimised Bokmål	target_lang="nob", style="reading_optimised"
Norwegian speech	Verbatim Bokmål	target_lang="nob", style="verbatim"
Norwegian speech	Written Nynorsk	target_lang="nno", either style
Norwegian speech	English translation	target_lang="eng", either style
Audio plus relevant vocabulary	The same task, with optional keyword context	keywords="metoprolol Tromsø Nordlandssykehuset"

Nynorsk and English training targets generally reuse Norwegian source audio; these are output-language controls, not evidence of an independently trained English-speech corpus. The original model's other languages have not been systematically revalidated after this adaptation.

Verbatim aims to retain spoken repetitions, repairs and hesitations. It does not guarantee exact reproduction of every disfluency. Reading-optimised output aims at readable written text; it is not summarisation. There is no hard-coded rule to cap repetitions at three. Neither <summary> nor the reserved <reading_optimised> token was used as a training style in this run.

Installation and gated access

Use a fresh Python 3.12 environment with a suitable CUDA-enabled GPU. The release examples target qwen-asr==0.0.6 / Transformers 4.57.6. That package's vLLM extra pins vLLM 0.14.0; use a compatible CUDA/PyTorch environment rather than upgrading individual components independently.

python -m pip install "qwen-asr==0.0.6" huggingface_hub soundfile scipy
hf auth login
hf download NbAiLab/nb-asr-qwen17b-bokfink021-beta --local-dir ./nb-asr-qwen17b-bokfink021-beta
cd nb-asr-qwen17b-bokfink021-beta

Request access through the NB-ASR beta collection. Once approved, authenticate with an HF token that can read this gated repository. Do not put tokens in source files, notebooks, shared logs or command-line examples. For reproducible deployment, download a fixed Hub commit with --revision; the model's release provenance is included in release_provenance.json.

The accompanying inference.py is a small, optional local helper. Read it before running it. It does not install custom model implementations or require trust_remote_code=True; Qwen classes come from the installed qwen-asr package.

Prompt contract: three independent controls

Always load the tokenizer, processor and chat_template.jinja from this repository. The custom tokenizer remaps reserved slots without changing the embedding dimensions. Do not substitute the original Qwen tokenizer or one from another NB-ASR release.

Control	Where it belongs	Exact content
Contextual keywords	SYSTEM message	Plain space-separated vocabulary, or ""
Output style	USER text, after the audio	"" for reading; "\n<verbatim>" for verbatim
Output language	Fixed ASSISTANT generation prefix	One of the headers below

Target code	Assistant prefix
nob	language Norwegian<asr_text>
nno	language Norwegian Nynorsk<asr_text>
eng	language English<asr_text>

The conceptual serialisation for keyword-conditioned verbatim Bokmål is:

<|im_start|>system
metoprolol Tromsø Nordlandssykehuset<|im_end|>
<|im_start|>user
<|audio_start|><|audio_pad|><|audio_end|>
<verbatim><|im_end|>
<|im_start|>assistant
language Norwegian<asr_text>

Use the processor to create this prefix; do not hand-tokenise the audio markers. The generated continuation is the transcript only. Leave SYSTEM empty when no keywords are needed. Do not place <verbatim> in the system message or put nob, nno, eng into the user message as instructions. Do not substitute Norwegian Bokmål for the trained Norwegian header.

The upstream Qwen3ASRModel.transcribe(language=...) validates against its original language list, which does not contain our Norwegian selectors. Also, its context parameter supplies SYSTEM text: context="<verbatim>" is wrong for this checkpoint. The helper uses explicit processor prompts and the ordinary Transformers/vLLM generation backends, avoiding both pitfalls without globally patching the upstream library.

Optional keyword boosting

Supply a short list of plausible names/terms, joined by spaces, preserving multiword phrases. No Vocabulary: prefix, comma-separated list, true/distractor labels or instructions are needed. For example:

metoprolol Eliquis Tromsø Nordlandssykehuset

Keywords are hints, not text that must be inserted. Most training rows had no keywords: 96.96% empty SYSTEM context; 1.70% true-plus-distractor context; 0.85% distractors only; 0.48% true keywords only. The contextual feature has not been separately benchmarked for this beta release. Prefer ordinary no-context ASR unless useful vocabulary is actually available.

Transformers: quickstart

The following commands use the downloaded helper and local checkpoint:

# Reading-optimised Bokmål
python inference.py /path/to/audio.wav --model . --target-lang nob

# Verbatim Bokmål
python inference.py /path/to/audio.wav --model . --target-lang nob --style verbatim

# Written Nynorsk
python inference.py /path/to/audio.wav --model . --target-lang nno --style verbatim

# English translation of Norwegian speech
python inference.py /path/to/audio.wav --model . --target-lang eng

# Verbatim with optional keyword context
python inference.py /path/to/audio.wav --model . --style verbatim \
  --keywords "metoprolol Eliquis Tromsø Nordlandssykehuset"

Output is JSON containing text, target_lang, style, duration_seconds, truncated and optional timestamps. The helper downmixes to mono and resamples to 16 kHz without normalising away quiet voices or applying VAD. It uses BF16, greedy decoding and explicitly enables the generation KV cache; this overrides the checkpoint's training-time use_cache=false setting without changing weights.

For Python integration, run from the downloaded directory:

from inference import BokfinkASR

model = BokfinkASR(model=".", backend="transformers", device="cuda:0")
result = model.transcribe(
    "/path/to/audio.wav",
    target_lang="nob",
    style="verbatim",
    keywords="",  # SYSTEM vocabulary; not the style tag
)
print(result["text"])

vLLM: transcription

Install the supported vLLM extra in a suitable environment:

python -m pip install "qwen-asr[vllm]==0.0.6"
CUDA_VISIBLE_DEVICES=0 python inference.py /path/to/audio.wav \
  --model . --backend vllm --target-lang nob --style verbatim \
  --gpu-memory-utilization 0.60 --max-model-len 32768 --max-new-tokens 4096

For reading-optimised output, omit --style verbatim. Nynorsk, English and keywords use exactly the same switches as the Transformers example.

Programmatic vLLM entry points must use the multiprocessing main guard:

from inference import BokfinkASR

if __name__ == "__main__":
    model = BokfinkASR(
        model=".", backend="vllm", gpu_memory_utilization=0.60,
        max_model_len=32768, max_new_tokens=4096,
    )
    print(model.transcribe("/path/to/audio.wav", target_lang="nob", style="verbatim"))

This is offline vLLM inference with the same explicit prompt contract, not an OpenAI-compatible HTTP server. The helper constructs requests with rendered prompt and multi_modal_data={"audio": [waveform]}. For a separate serving integration, preserve all three controls above: generic transcription APIs or chat templates that omit the assistant-language prefix can change the task.

The example limits concurrent sequences to one to give a conservative starting point for long audio. Tune batch/concurrency and memory utilisation on your own hardware; no universal minimum VRAM or throughput claim is made here.

vLLM or Transformers together with the forced aligner

Use the Norwegian-adapted companion NbAiLab/nb-asr-beta-qwen06b-08052026-forced-aligner. The helper pins its revision to 6ddd1268709255a14f576b4db898942eb24b8e38. It is a separate model, not part of the ASR weights. The aligner runs through Transformers even when ASR runs through vLLM; reserve GPU memory for both, or place them on separate devices. Separate HF access may be required for it.

For a short recording:

CUDA_VISIBLE_DEVICES=0 python inference.py /path/to/audio.wav \
  --model . --backend vllm --target-lang nob --style verbatim \
  --gpu-memory-utilization 0.60 --timestamps

Or in Python:

from inference import BokfinkASR

if __name__ == "__main__":
    model = BokfinkASR(model=".", backend="vllm", gpu_memory_utilization=0.60)
    model.load_aligner(device="cuda:0")
    result = model.transcribe("/path/to/audio.wav", style="verbatim", timestamps=True)
    print(result["text"])
    for word in result["timestamps"]:
        print(word["text"], word["start"], word["end"])

For ASR on GPU 0 and alignment on GPU 1, expose both GPUs, set CUDA_VISIBLE_DEVICES=0,1, and use --aligner-device cuda:1. vLLM uses its normal visible-device selection; the example uses one tensor-parallel ASR GPU. The same alignment API works with backend="transformers".

Important alignment boundaries:

• The helper conservatively limits a combined ASR+alignment call to 180 s, matching the installed upstream timestamp pipeline. For longer recordings, split at sensible speech boundaries, transcribe and align each segment, then add each segment's start offset to its word timestamps. It never silently truncates a 20-minute file to fit the aligner.
• Prefer verbatim text for acoustic alignment. Rewritten or omitted words in reading-optimised output need not have one-to-one spans in the audio.
• Nynorsk alignment is experimental; different written forms can affect timing. Do not align an English translation directly to Norwegian audio; the helper rejects this combination. Transcribe/align the spoken words first and handle translation timing as a separate task.
• Empty ASR outputs return an empty timestamp list. Forced alignment does not verify that a transcript is correct or prove that the words were spoken.

Long recordings and generation limits

Training includes short clips and long/sparse-speech examples up to 1,200 s (20 minutes). This is the supported input ceiling of the preparation/example path, not a guarantee of accurate 20-minute transcription. The small natural long-form evaluation cohort is much shorter than 20 minutes; long recordings need independent testing. Streaming behaviour has not been validated for this release helper.

The helper rejects audio over 1,200 s rather than cropping it. It disables processor truncation and defaults to 4,096 generated tokens. Dense long speech may require a larger token limit and context budget. If generation reaches the limit, it warns and returns truncated=true; do not mistake this for a complete transcript. Check memory and quality when increasing limits.

Training and data

Full-parameter BF16 training started fresh from NbAiLab/Qwen3-ASR-1.7B-custom-tokenizer; it did not continue from the earlier Bokfink02 beta or the Lunde checkpoints. The corpus refresh targets punctuation and label-quality problems, adds the rarewords90 synthetic source and uses the v4 augmentation recipe. These changes are not isolated causal ablations; the benchmark tables describe these weights.

Setting	Value
Internal version	0.2.1-beta, Qwen3-ASR 1.7B family
Starting revision	7b5ea69e9b39b28f1b1221662959e02a1805f5a5
Final checkpoint	26,188 optimizer steps; 2.0 row epochs
Prepared corpus	18,208,790 target rows across 21 sources
Hardware	64 GPUs across 16 nodes, independently for each model
Optimizer / LR	Fused AdamW; peak 1e-4 for both sizes
LR schedule	3% warmup (786 updates); cosine decay to 1e-5 at the final step
Batching	Row epochs; up to 24 short or one long example per GPU; gradient accumulation 1
Input/token budget	Up to 1,200 s audio; 32,768 input tokens; overlength inputs are not silently token-truncated
Loss	Global supervised-token mean; prefix and padding masked
Data order	Shuffled row epochs, seed 42; deterministic contextual keyword selections
Audio augmentation	Probability 0.50, noisy_v4; phone/radio/Teams/MELPe weights 1/1/1.5/1
Speed perturbation / BPE dropout	Neither enabled
Evaluation/save cadence	Step 0, every 2k updates, and final 26,188; separate single-GPU evaluators

The source inventory below records the releases used for these weights. Current dataset main branches may have changed since training. Exact pinned revisions are in release_provenance.json. Rows are training targets, not unique audio: languages and styles can share recordings. Six-way sources expand into three output languages multiplied by two styles, not six speakers.

Source	Target language	Output style
consensus-reading-nob	nob	reading / empty
consensus-verbatim-nob	nob	verbatim
nemopseudo-reading-nob	nob	reading / empty
nemopseudo-verbatim-nob	nob	verbatim
synthetic-numbers-nob	nob	reading / empty
synthetic-rarewords10-nob	nob	reading / empty
consensus-reading-nno	nno	reading / empty
consensus-verbatim-nno	nno	verbatim
nemopseudo-reading-nno	nno	reading / empty
nemopseudo-verbatim-nno	nno	verbatim
consensus-reading-eng	eng	reading / empty
consensus-verbatim-eng	eng	verbatim
nemopseudo-reading-eng	eng	reading / empty
nemopseudo-verbatim-eng	eng	verbatim
nocaption	nob	reading / empty
background-noise-20pct	nob	reading / empty
stortinget-long	nob	reading / empty
dense-sixway-long	nob / nno / eng	both styles (per row)
sparse-sixway-long	nob / nno / eng	both styles (per row)
noise-only-long	nob	reading / empty
synthetic-rarewords90-nob	nob	reading / empty

Short sources use their text targets, subject to the source-specific preparation rules. Norwegian sources include upstream NormForm normalization. The six-way long sources use their per-language/style text fields. Empty targets in recorded no-caption, the 20% background-noise dataset, and long-noise sources are intentional; the full background-noise dataset is not also included. Long noise can be cropped to the 20-minute input ceiling. Pseudo-labels, synthetic speech, and translated targets may contain errors. No summary task was trained.

Final-checkpoint benchmarks

All results below are from these exact final 26,188-step weights, not an earlier checkpoint. Both models use the same pinned references, corrected style template, greedy decoding, empty system context and deterministic tokenization. The complete numeric tables and scoring provenance are in benchmarks.json.

WER and CER are percentages, lower is better. Text is casefolded; punctuation and non-alphanumeric characters are removed and whitespace collapsed. Thus these scores do not measure punctuation/capitalization quality. WER is capped at 100% per clip before reference-word-weighted aggregation; CER is uncapped and reference-character-weighted. Displayed values are rounded only after scoring.

Overview: WER

Evaluation	0.6B	1.7B — this model
Short-form macro, all languages/styles	14.47	13.73
Bokmål macro	8.55	8.48
Nynorsk macro	12.40	12.27
English-translation macro	22.47	20.44
NST Bokmål, both styles	4.82	4.24
Natural long-form Bokmål, verbatim	23.90	18.27
Noisy NST, five-condition macro	5.08	4.06

Short macro averages the four domains (audiobooks, NRK, NST, Stortinget) and both styles equally; the overall macro also weights the three languages equally. This is different from pooling all reference words across domains. Each domain has 100 unique audio clips; 400 Norwegian short recordings are reused across languages/styles. Fleurs and NB-tale are excluded from short-form aggregates.

Short-form detail: WER / CER

Each cell below is WER / CER, computed separately for its 100 examples. Reading uses no user style tag; verbatim uses <verbatim> in USER. Output language is fixed by the assistant prefix described above. Nynorsk and English references are model-translated and have not been fully human-reviewed; English WER/CER is exact-reference translation agreement, not a general translation quality metric.

Bokmål

Domain	Style	0.6B	1.7B — this model
audiobooks	reading_optimised	8.95 / 3.39	8.11 / 3.11
audiobooks	verbatim	8.63 / 3.10	8.74 / 3.21
nrk	reading_optimised	10.12 / 6.62	10.92 / 6.67
nrk	verbatim	10.82 / 6.47	11.47 / 6.71
nst	reading_optimised	4.38 / 1.11	4.02 / 0.88
nst	verbatim	5.27 / 1.38	4.46 / 1.05
stortinget	reading_optimised	10.53 / 5.74	10.12 / 5.54
stortinget	verbatim	9.73 / 4.55	10.01 / 4.71

Nynorsk

Domain	Style	0.6B	1.7B — this model
audiobooks	reading_optimised	14.21 / 5.43	15.27 / 5.90
audiobooks	verbatim	14.85 / 5.52	15.58 / 5.95
nrk	reading_optimised	13.52 / 7.72	13.59 / 11.96
nrk	verbatim	13.05 / 7.00	13.22 / 6.97
nst	reading_optimised	6.26 / 1.74	5.56 / 1.68
nst	verbatim	6.61 / 2.01	5.38 / 1.28
stortinget	reading_optimised	15.20 / 7.86	14.39 / 7.17
stortinget	verbatim	15.47 / 6.88	15.16 / 6.67

English translation

Domain	Style	0.6B	1.7B — this model
audiobooks	reading_optimised	29.26 / 20.92	26.21 / 19.05
audiobooks	verbatim	30.41 / 21.25	27.53 / 19.58
nrk	reading_optimised	25.59 / 18.79	24.35 / 18.01
nrk	verbatim	27.68 / 20.45	25.70 / 18.79
nst	reading_optimised	8.59 / 7.53	7.34 / 5.93
nst	verbatim	7.70 / 6.73	8.16 / 5.98
stortinget	reading_optimised	24.86 / 18.09	22.44 / 16.58
stortinget	verbatim	25.67 / 20.57	21.79 / 15.64

Noisy NST: reading-optimised WER / CER

The same 100 underlying NST clips are evaluated in each recording condition. Noisy and ordinary NST are separate cohorts and should not be assumed identical.

Condition	0.6B	1.7B — this model
studio	2.34 / 0.69	2.58 / 0.75
room_far	6.29 / 2.40	3.55 / 1.13
teams	5.24 / 2.09	4.67 / 1.72
melpe2400	4.35 / 1.52	3.71 / 1.18
melpe1200	7.17 / 2.81	5.80 / 2.25

Natural long-form: verbatim WER / CER

Cohort	0.6B	1.7B — this model
NB-tale, seven recordings	23.90 / 14.84	18.27 / 9.47

Seven recordings are a small diagnostic cohort and do not validate the full 20-minute input range. Long reading-optimised scores are omitted because known reference omissions/rewrites make them unreliable for model comparison.

Empty-reference audio: percentage with text / mean generated tokens

These measures describe outputs on audio whose references are empty. They are not confirmed hallucination rates: listening review found faint genuine voices in some recorded examples. The corrected recorded cohort has 94 clips; synthetic noise has 100. Both styles are tested with a 100-token generation cap. Token counts use the same model-native tokenizer across the two models; these are not word counts. WER/CER is not reported for empty references.

Cohort	Style	0.6B	1.7B — this model
nocaptions	reading_optimised	0.00% / 0.000	0.00% / 0.000
nocaptions	verbatim	3.19% / 0.160	5.32% / 0.468
nocaptions_synthetic	reading_optimised	0.00% / 0.000	0.00% / 0.000
nocaptions_synthetic	verbatim	0.00% / 0.000	0.00% / 0.000

Evaluation dataset revisions

Dataset	Split	Revision
NbAiLab/nb-asr-evaluering-sprakradet	eval	9f1bbf7099d840e50c2df3e44873b13b4436d4cc
NbAiLab/nb-asr-evaluering-sprakradet-nno	eval	2992e5a9ae2749c64a84173419ad571bd5583eb9
NbAiLab/nb-asr-evaluering-sprakradet-eng	eval	5ef6989d2f37a131a1f6ea157c8c73256e5d87b3
NbAiLab/nb-asr-noisy-recordings	eval	29a4ef4f3fa80835cacf649088f3c73f61def062
NbAiLab/nb-asr-evaluering-sprakradet	eval	7b57c04f30ecf9bd0f70513aca2802dfec0f00bd

The no-speech subset uses the audited deletion-only update of the primary dataset; other primary references remain pinned to their original revision. The source prediction/metric hashes and no-speech selection checksum are in benchmarks.json. No evaluation audio or transcript reports are distributed in this model repository.

Known limits

• Repetition, omissions, wrong names and early stopping remain possible, including isolated errors on long recordings. Mean WER does not rule these out.
• Verbatim does not guarantee every hesitation; this release has not received a separate hesitation-boost post-training stage.
• Punctuation, dialect/speaker balance, rare names, contextual boosting, sparse speech and the full 20-minute range need independent assessment. No complete recording-level training/evaluation decontamination claim is made.
• Background-speech rejection and invented content require listening-based assessment; empty-reference output alone does not distinguish them.
• Streaming and the newly documented vLLM-plus-aligner combination have not been validated end-to-end as part of this release. Preserve the explicit prompts when integrating another inference interface.

Internal version and release verification

Maintainer reference: 0.2.1-beta, Qwen3-ASR 1.7B

Internal production run main17b, Slurm 2596524; final checkpoint-26188, epoch 2.0, augmentation 0.50/noisy_v4, BPE dropout disabled. This is the completed two-epoch Bokfink021 model, not the old 28k Bokfink02 release and not the separate BPE experiment. The training launcher reported a shutdown parsing failure after completing the checkpoint; final model inference and full evaluation completed successfully. No further optimizer updates were applied during release preparation.

Weights, config, tokenizer and processor files come byte-for-byte from the final checkpoint. The obsolete chat_template.json, which can shadow the correct user-style template, is deliberately excluded; chat_template.jinja is retained and the release prompt serialization is checked. Token IDs, vocabulary/embedding dimensions, disabled tokenizer dropout, complete BF16 safetensors payload and source-base vocabulary/merges are verified. Release files and hashes, training source revisions and GPU smoke results are recorded in release_provenance.json. The helper and model card are release support files, not weight modifications. Optimizer, scheduler, RNG, Trainer state, training arguments, manifests, raw audio, transcripts, logs and credentials are excluded. Full recovery state remains in internal storage. All uploaded file identities and public/gated collection membership are checked after publication.

References

• Qwen3-ASR implementation and installation
• Upstream architecture
• Custom-tokenizer starting model
• Bokfink02 beta usage reference
• Norwegian companion aligner

Developed within NB-ASR at the National Library of Norway.