Chord Recognition with Deep Learning

TL;DR

This work investigates why automatic chord recognition using deep learning has stagnated and conducts a comprehensive analysis of a standard CRNN baseline, identifying that performance is bottlenecked by rare-chord data, transition timing, and limited context. It systematically tests improvements including weighted and structured losses, decoding with HMM/CRF smoothing, pitch augmentation, generative features, synthetic data, and beat-synchronous predictions. Key findings show pitch augmentation and decoding improve accuracy and interpretability, while generative features offer little benefit over CQTs; beat-synchronous outputs yield the strongest qualitative gains, with a perfect-beats setup achieving the highest mirex score reported in the literature. The work also demonstrates that calibrating data distribution via synthetic datasets and focused loss design can modestly boost class-wise metrics, suggesting a path forward for more robust and musically meaningful chord recognition systems. Overall, the study provides a solid foundation for future exploration into beat-aware chord recognition and data-driven methods to address long-tail chord distributions.

Abstract

Progress in automatic chord recognition has been slow since the advent of deep learning in the field. To understand why, I conduct experiments on existing methods and test hypotheses enabled by recent developments in generative models. Findings show that chord classifiers perform poorly on rare chords and that pitch augmentation boosts accuracy. Features extracted from generative models do not help and synthetic data presents an exciting avenue for future work. I conclude by improving the interpretability of model outputs with beat detection, reporting some of the best results in the field and providing qualitative analysis. Much work remains to solve automatic chord recognition, but I hope this thesis will chart a path for others to try.

Figures (25)

• Figure 1: CQT of 'Girls Just Wanna Have Fun' by Cyndi Lauper. We can see the log-spaced frequency bins on the y-axis. There is clear structure and repetition in the song, particularly in the lower frequencies, which can be attributed to a regular drum groove. Such structure gives an idea of the patterns a machine learning model may look for to identify chords.
• Figure 2: Histograms of the maximum cross-correlation lags and the difference in length between the audio and chord annotations. Both show results close to $0$, suggesting good alignment between audio and annotations.
• Figure 3: Chord distributions over qualities (left) and roots (right) in the pop dataset. The plots show the raw counts in frames and the duration in seconds for each chord root/quality. Note that the y-axis over qualities is in a logarithmic scale. The qualities are very imbalanced, with maj as the most popular. Conversely, roots are relatively balanced.
• Figure 4: Row-normalised confusion matrices over qualities of the CRNN model. Rows are ordered by frequency of chord quality. We observe that the model struggles with the imbalanced distribution. It frequently confuses dim7 and dim qualities, consistently predicts maj for sus2, sus4, maj6, maj7 and struggles with the rare qualities like minmaj7 and aug.
• Figure 5: Histogram of accuracies and mirex scores over songs in the validation set. Accuracies are mixed, with $15\%$ of songs below $40\%$, and $69\%$ between $0.4$ and $0.8$. However, with the more generous mirex metric, we find that there are almost no songs below a score of $40\%$ and only $7\%$ below $0.6$. This suggests that many of the mistakes the model makes are small, like predicting C:maj instead of C:maj7. The very low outliers in the mirex score were found to be songs with incorrect annotations found in Section \ref{['sec:data-integrity']}.