Efficient and Generalizable Speaker Diarization via Structured Pruning of Self-Supervised Models
Paper • 2506.18623 • Published
Non-commercial research use only
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These files are Core ML conversions of DiariZen (BUT-FIT/diarizen-wavlm-base-s80-md) by BUT Speech@FIT. The DiariZen weights are licensed under CC BY-NC 4.0, and BUT Speech@FIT states that by downloading them you agree to use them for research and academic purposes only. The converted files carry the same terms: no commercial use, and you must give appropriate credit to DiariZen (BUT Speech@FIT) and to the WavLM, pyannote.audio and WeSpeaker work it builds on, link to the license, and indicate changes. License: https://creativecommons.org/licenses/by-nc/4.0/
Log in or Sign Up to review the conditions and access this model content.
DiariZen (WavLM Base+ s80) for Core ML
The neural parts of BUT Speech@FIT's DiariZen speaker diarization pipeline, checkpoint BUT-FIT/diarizen-wavlm-base-s80-md, converted to Core ML for Apple Silicon. The checkpoint is a WavLM Base+ front end, structurally pruned to 80% sparsity, under a Conformer with a powerset head; DiariZen pairs it with a WeSpeaker ResNet34 embedding and VBx clustering, with no cap on the number of speakers.
These are the files the Swift runtime diarizen-apple loads. That project has the rest of the pipeline in Swift (sliding window, powerset decoding, Kaldi filterbank, VBx, reconstruction to RTTM) and the steps to convert the models yourself; this repository saves you the conversion.
Non-commercial use only. See the license section.
Files
Segmentation.mlpackage/ 63 MB, float16: WavLM Base+ s80 + Conformer, powerset head
Embedding.mlpackage/ 13 MB, float16: WeSpeaker ResNet34-LM with masked statistics pooling
plda_transform.json 1.0 MB: DiariZen's VBx PLDA, eigendecomposition folded in
conversion/ the export scripts (MIT, from diarizen-apple)
LICENSE CC BY-NC 4.0 legal code
Accept the terms on this page, log in (hf auth login), download, and point
the runtime at the folder from a clone of diarizen-apple:
hf download nikioth/diarizen-wavlm-base-s80-md-coreml --local-dir models/base
swift run -c release diarizen-cli meeting.wav --models models/base --output meeting.rttm
Model inputs and outputs
| Model | Input | Output |
|---|---|---|
Segmentation.mlpackage |
waveform (1, 1, 256000): one 16 s chunk, mono 16 kHz, samples in [-1, 1] |
segmentation (1, 799, 11): per-frame powerset log-probabilities |
Embedding.mlpackage |
fbank (B, 1598, 80) and weights (B, 799), B = 1 to 64 |
embedding (B, 256) |
Arrays are declared Float16; Core ML converts Float32 input.
Segmentation frames are 25 ms every 20 ms (frame f starts at about
f * 0.02 - 0.0075 s within the chunk). The 11 powerset classes encode up to
4 local speakers with at most 2 active in one frame: the empty set, the 4
single speakers, then the 6 pairs, in pyannote's powerset order. Take the
argmax per frame and map it to its speaker set. DiariZen slides the chunk
with a 1.6 s step, applies an 11-frame median filter to the decoded
activities, and aggregates overlapping chunks.
The embedding input is WeSpeaker's frontend: torchaudio.compliance.kaldi.fbank
on samples scaled by 32768 (80 mel bins, 25 ms Hamming frames every 10 ms,
DC removal, pre-emphasis 0.97, no dither) with the mean over the 1598 frames
subtracted, computed on the same 16 s chunk. weights marks the frames of one
local speaker (pyannote uses the speaker's non-overlapped frames when there are
enough of them). The ResNet34 is the same network as
nikioth/wespeaker-resnet34-lm-coreml,
which documents the frontend in detail.
plda_transform.json holds mean1, lda, mean2, plda_mu, plda_tr
and plda_psi as {"shape", "data"} float64 arrays: the output of VBx's
vbx_setup for DiariZen's plda/ files. A 256-d embedding x becomes VBx
features with
y = sqrt(128) * l2(lda.T @ (sqrt(256) * l2(x - mean1)) - mean2) and
features = (y - plda_mu) @ plda_tr.T; plda_psi is VBx's Phi. DiariZen's
clustering settings for this checkpoint: AHC initialization at threshold 0.6,
VBx with Fa 0.07, Fb 0.8, LDA dimension 128, 20 iterations.
Accuracy
Measured with the diarizen-apple Swift runtime using these files, on an
Apple M2 Pro, against human references (pyannote/AMI-diarization-setup,
pyannote.metrics, collar 0 unless noted):
| Test | This conversion | Original PyTorch pipeline |
|---|---|---|
| AMI-SDM test set, 16 meetings, corpus DER | 15.79% | 15.8% (published by DiariZen) |
| AMI EN2002a (35.7 min, 4 speakers), DER | 21.16% | 21.10% |
| AMI EN2002a, DER between the two outputs | 1.90% | |
| VoxConverse files with 12 to 21 speakers (31 files), collar 0.25 | 7.24% (finds 8 to 23 speakers) | |
| 30 s parity clip, DER against the PyTorch RTTM | 0.495% |
Stage parity: the Core ML segmentation agrees with PyTorch on 99.87% of frame argmaxes (largest probability difference 2.1e-2, mean 7.8e-4); the remaining differences sit in the deep-negative log-probability tail and appear in float32 too, so they come from runtime numerics, not float16. The embedding stage matches PyTorch at cosine 0.99991 with the Swift filterbank in front. Swift's Core ML output is bit-identical to coremltools' on the same input.
Speed
On an Apple M2 Pro (32 GB), CPU and GPU, model loading excluded:
| Time | |
|---|---|
| Segmentation, one 16 s chunk (coremltools) | 28 ms |
| Full Swift pipeline, 30 s clip | 1.12 s (real-time factor 0.037), peak memory 298 MB |
| Full Swift pipeline, 35.7 min meeting | 151 s (real-time factor 0.071) |
| PyTorch pipeline on the CPU, same meeting | 3042 s |
Embedding extraction takes most of the time; it runs batched (32 crops per Core ML call).
Limitations
- Do not use the Neural Engine for the segmentation model. With compute units
.allthe first prediction deadlocks; with.cpuAndNeuralEngineCore ML places none of its 643 ops on the Neural Engine and segmentation runs about 12 times slower than on the GPU. Use.cpuAndGPU. The embedding ResNet can use the Neural Engine, with little gain. - Fixed shapes: one 16 s chunk per segmentation call. Pad the last chunk with zeros as pyannote does.
- Benchmarks come from one machine, one 30 s clip and one full meeting for speed. Accuracy figures are the public AMI and VoxConverse sets above.
- The training data is far-field meeting and conversational audio (AMI, AISHELL-4, AliMeeting, NOTSOFAR-1, MSDWild, DIHARD3, RAMC, VoxConverse); other conditions were not measured here.
Converting it yourself
From a clone of diarizen-apple,
with its Python 3.11 environment (requirements.txt, torch 2.1.1,
coremltools 9.0, DiariZen installed from source):
python conversion/export_coreml.py --model BUT-FIT/diarizen-wavlm-base-s80-md --out out
python conversion/export_embedding_coreml.py --model BUT-FIT/diarizen-wavlm-base-s80-md --out out
python conversion/export_plda.py --model BUT-FIT/diarizen-wavlm-base-s80-md --out out/plda_transform.json
conversion/ here is a copy of those scripts (diarizen-apple commit
ca61fa0). The published packages were exported in September 2026 with an
earlier revision of the same scripts. plda_transform.json is reproduced byte
for byte by applying vbx_setup to the checkpoint's plda/plda.npz and
plda/xvec_transform.npz at revision e73cd10d.
License and attribution
The repository as a whole is released under
CC BY-NC 4.0 (LICENSE):
non-commercial research and academic use only, with attribution.
Segmentation.mlpackage: converted from DiariZen'sBUT-FIT/diarizen-wavlm-base-s80-mdby BUT Speech@FIT, CC BY-NC 4.0 (some of its training sets are research-only). It contains a pruned WavLM Base+ (Microsoft) and pyannote.audio's powerset formulation.Embedding.mlpackage: WeSpeaker ResNet34-LM trained on VoxCeleb, as packaged by pyannote (pyannote/wespeaker-voxceleb-resnet34-LM), CC BY 4.0. It is the embedding DiariZen uses; the same file under its own terms is in nikioth/wespeaker-resnet34-lm-coreml.plda_transform.json: derived from the checkpoint'splda/files, a PLDA and x-vector transform trained by BUT Speech@FIT on VoxCeleb2, which BUT licenses under CC BY 4.0.conversion/: MIT (conversion/LICENSE).
Changes made by nikioth: the segmentation and embedding networks were
converted from PyTorch to Core ML ML Programs (float16, fixed 16 s chunk, two
export-only patches that are identity at inference, see
conversion/patches/README.md); the PLDA eigendecomposition was precomputed
into one JSON file. No weights were retrained. Provided as is, without
warranties of any kind.
Citation
@inproceedings{han2025leveraging,
title = {Leveraging self-supervised learning for speaker diarization},
author = {Han, Jiangyu and Landini, Federico and Rohdin, Johan and Silnova, Anna and
Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
booktitle = {Proc. ICASSP},
year = {2025}
}
@article{han2025fine,
title = {Fine-tune Before Structured Pruning: Towards Compact and Accurate
Self-Supervised Models for Speaker Diarization},
author = {Han, Jiangyu and Landini, Federico and Rohdin, Johan and Silnova, Anna and
Diez, Mireia and Cernocky, Jan and Burget, Lukas},
journal = {arXiv preprint arXiv:2505.24111},
year = {2025}
}
@article{han2025efficient,
title = {Efficient and Generalizable Speaker Diarization via Structured Pruning of
Self-Supervised Models},
author = {Han, Jiangyu and P{\'a}lka, Petr and Delcroix, Marc and Landini, Federico and
Rohdin, Johan and Cernock{\`y}, Jan and Burget, Luk{\'a}{\v{s}}},
journal = {arXiv preprint arXiv:2506.18623},
year = {2025}
}
@article{chen2022wavlm,
title = {WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing},
author = {Chen, Sanyuan and Wang, Chengyi and Chen, Zhengyang and Wu, Yu and Liu, Shujie and
Chen, Zhuo and Li, Jinyu and Kanda, Naoyuki and Yoshioka, Takuya and Xiao, Xiong and
Wu, Jian and Zhou, Long and Ren, Shuo and Qian, Yanmin and Qian, Yao and Wu, Jian and
Zeng, Michael and Yu, Xiangzhan and Wei, Furu},
journal = {IEEE Journal of Selected Topics in Signal Processing},
volume = {16},
number = {6},
pages = {1505--1518},
year = {2022}
}
@inproceedings{plaquet2023powerset,
title = {Powerset multi-class cross entropy loss for neural speaker diarization},
author = {Plaquet, Alexis and Bredin, Herv{\'e}},
booktitle = {Proc. INTERSPEECH 2023},
year = {2023}
}
@inproceedings{bredin2023pyannote,
title = {pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe},
author = {Bredin, Herv{\'e}},
booktitle = {Proc. INTERSPEECH 2023},
pages = {1983--1987},
year = {2023}
}
@inproceedings{wang2023wespeaker,
title = {Wespeaker: A research and production oriented speaker embedding learning toolkit},
author = {Wang, Hongji and Liang, Chengdong and Wang, Shuai and Chen, Zhengyang and
Zhang, Binbin and Xiang, Xu and Deng, Yanlei and Qian, Yanmin},
booktitle = {ICASSP 2023 - IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2023}
}
@article{landini2022vbx,
title = {Bayesian HMM clustering of x-vector sequences (VBx) in speaker diarization:
theory, implementation and analysis on standard tasks},
author = {Landini, Federico and Profant, J{\'a}n and Diez, Mireia and Burget, Luk{\'a}{\v{s}}},
journal = {Computer Speech \& Language},
volume = {71},
pages = {101254},
year = {2022}
}
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for nikioth/diarizen-wavlm-base-s80-md-coreml
Base model
BUT-FIT/diarizen-wavlm-base-s80-md