---
license: apache-2.0
library_name: transformers
pipeline_tag: audio-classification
inference: false
tags:
- audio
- audio-classification
---

# MATPAC++ (AudioSet fine-tuned encoder)

**MATPAC** (Masked latent Prediction And Classification) is a self-supervised audio
and music encoder. This repo hosts the **MATPAC++ encoder, self-supervised then
fine-tuned on AudioSet, encoder only** (`matpac_plus_as_48_1_map_enconly.pt`), a
`transformers`-native, `trust_remote_code` port of the official inference code
([aurianworld/matpac](https://github.com/aurianworld/matpac), folder
`inference_matpac`).

- 📝 [MATPAC: Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning](https://arxiv.org/abs/2502.12031)
- 📝 [MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning](https://arxiv.org/pdf/2508.12709)

If you want ready-made AudioSet class predictions instead of embeddings, use
[matpac_audioset_finetune_classifier](https://huggingface.co/auriankelen/matpac_audioset_finetune_classifier)
(same fine-tuning, plus the 527-way classification head).

## The MATPAC family

| Repo | Trained on | Output | Notes |
| --- | --- | --- | --- |
| [matpac_general_audio](https://huggingface.co/auriankelen/matpac_general_audio) | General audio (AudioSet) | 3840-d embeddings | Default general-purpose audio encoder |
| [matpac_music](https://huggingface.co/auriankelen/matpac_music) | Music | 3840-d embeddings | Music-specialized encoder |
| **matpac_audioset_finetune_encoder** (this repo) | AudioSet (fine-tuned) | 3840-d embeddings | Fine-tuned encoder, no classification head |
| [matpac_audioset_finetune_classifier](https://huggingface.co/auriankelen/matpac_audioset_finetune_classifier) | AudioSet (fine-tuned) | 527-way logits | Ready-to-use AudioSet tagger |

## Usage

```python
from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
import torchaudio

model = AutoModel.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)
processor = Wav2Vec2FeatureExtractor.from_pretrained("auriankelen/matpac_audioset_finetune_encoder", trust_remote_code=True)

waveform, sr = torchaudio.load("my_file.wav")
waveform = waveform.mean(dim=0)  # mono

resample_rate = processor.sampling_rate
if resample_rate != sr:
    waveform = torchaudio.transforms.Resample(sr, resample_rate)(waveform)

inputs = processor(waveform, sampling_rate=resample_rate, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state: (batch, 3840) -- mean-pooled over time by default
# outputs.hidden_states: tuple of 12 tensors, one per encoder layer, each (batch, 3840)
all_layer_hidden_states = torch.stack(outputs.hidden_states)
print(all_layer_hidden_states.shape)  # [12, batch, 3840]
```

### Keeping the time dimension / faster inference

```python
# Keep the time axis instead of mean-pooling it
outputs = model(**inputs, pull_time_dimension=False)
# outputs.last_hidden_state: (batch, time, 3840)

# Vectorized inference: faster on large batches, adds some padding so it is
# slightly less precise than the default ("precise", used for the paper's results)
outputs = model(**inputs, inference_type="fast")
```

## Requirements

This model needs a couple of packages beyond `transformers` to run its
`trust_remote_code` files: `torchaudio`, `einops`, and `timm==0.4.12` (the exact
`timm` version the checkpoints were exported with the internal naming of the ViT
block submodules must match for the weights to load correctly).

## Citation

```bibtex
@inproceedings{quelennec2025matpac,
  title={Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning},
  author={Quelennec, Aurian and Chouteau, Pierre and Peeters, Geoffroy and Essid, Slim},
  booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2025},
  url={https://ieeexplore.ieee.org/document/10887666},
  doi={10.1109/ICASSP49660.2025.10887666}
}

@article{quelennec2025matpacenhancedmaskedlatent,
  title={MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning},
  author={Aurian Quelennec and Pierre Chouteau and Geoffroy Peeters and Slim Essid},
  journal={arXiv preprint arXiv:2508.12709},
  year={2025},
  url={https://arxiv.org/abs/2508.12709}
}
```

## Credits

- [aurianworld/matpac](https://github.com/aurianworld/matpac) for the original training and inference code.
- [Fairseq](https://github.com/facebookresearch/fairseq) for the training framework.
- [M2D](https://github.com/nttcslab/m2d/tree/master) for the base of the inference code.
- [DINO](https://github.com/facebookresearch/dino) for the classification head.