JAAH chord benchmark

Updated 2026-09-16. All 113 recordings in the Jazz Audio-Aligned Harmony (JAAH) dataset are evaluated with fixed models and the same reference intervals. Scores are percentages; higher is better. No model is trained or tuned on JAAH for this comparison.

Results

Model	Maj/min
SheetSage1	39.77
madmom	51.24
ChordFormer	59.45
SheetSage2-Prober	62.94
SheetSage2-AR	64.50

AR exceeds ChordFormer by 5.06 percentage points in Maj/min recall. The AR-minus-Prober difference is 1.56 points. These are differences in observed point estimates for the evaluated models and recordings. Bold marks the highest point estimate in this comparison.

Evaluation protocol

Cohort: all 113 original recordings; no track or chorus selection, reference-beat inputs, test-time pitch augmentation, global timing shift, or JAAH parameter tuning. The audio is from the supplied recordings, not reconstructed spectrograms.
References: byte-preserved official chord labels at JAAH commit 6fec3f6, intersected with each WAV's actual time span. Shorter annotations are not extended. This removes 9.146654 seconds of trailing N labels beyond the audio, leaving 24,991.969346 seconds. Five tracks have overruns above 20 ms and 37 have smaller rounding differences. The original labels are preserved.
Metric: per-track mir_eval.chord.majmin recall. Reference segments outside the metric's vocabulary receive -1 and are excluded from that track's metric denominator; they are never converted to N. Eligible reference duration, including N, is 93.87%.
Aggregation: weight each track's recall by its complete evaluated reference span. This is the same aggregation as the other SheetSage2 chord benchmarks. Macro averages and recall pooled by metric-eligible duration are retained in the JSON under separate names and are not the main-table values.
SheetSage1: released Jukebox-based model, automatic beats, six-measure chunks with the existing four-/two-measure fallback, and native structured harmony outputs. The fixed settings match its other chord benchmarks.
madmom: released CNN chord features and CRF chord decoder with default 10-Hz decoding and major/minor/no-chord vocabulary. The fixed settings match its other chord benchmarks.
ChordFormer: five released Chords1217 fold models, component-probability ensemble followed by the official HMM, and the predeclared submission vocabulary, matching osu2017. On Chords1217 itself, each track instead uses only its held-out fold model. Source: official ChordFormer commit 3da11c0.
SheetSage2-Prober: fixed checkpoint, 24-kHz mono audio, 300-second windows, BF16, batch size one, native predicted beats with a fixed zero global offset; melody decoding disabled.
SheetSage2-AR: fixed paper checkpoint, 24-kHz mono audio, 300-second windows with 100-second overlap and predicted-prefix carry, BF16 autocast with FP32 weights, SDPA, greedy decoding, and a 5,120-token limit. Prompts: timestamp, downbeat_meter, chord_full. Two of 132 windows (blues_for_alice, lady_bird) reach the token limit and use the existing decoder's recovery for incomplete events. All 113 tracks are retained.

Using the untrimmed official annotation spans changes each main score by less than 0.03 percentage points. ChordFormer's other fixed vocabulary, full, is included only as an auxiliary result (59.11 Maj/min); the primary vocabulary was fixed before seeing JAAH scores. Independent rescoring with mir_eval 0.7 and 0.8.2 agreed on all 904 model/reference/track combinations.

A title-and-performer audit found no matches between JAAH and Chords1217. This does not establish absence from foundation-model pretraining or constitute an audio-fingerprint audit. Published JAAH results on selected choruses, rewritten labels, different vocabularies, or different train/test splits are not directly comparable to this full-collection evaluation. Only chord results are reported here.

Machine-readable results

Full-precision scores, reference variants, metric coverage, and checkpoint hashes
Per-track scores and prediction/reference SHA-256 hashes: 1,356 rows = 113 tracks × six model/vocabulary variants × two reference policies. Main results use reference_variant=audio_span and exclude ChordFormer_full_auxiliary.
Point estimates and paired differences
SheetSage2 paper preset and dataset prompts

Dataset citation

Vsevolod Eremenko, Emir Demirel, Baris Bozkurt, and Xavier Serra. “Audio-aligned jazz harmony dataset for automatic chord transcription and corpus-based research.” Proceedings of the 19th International Society for Music Information Retrieval Conference, 2018, pp. 483–490. DOI: 10.5281/zenodo.1291834.