Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Theory-Based Explainable Deep Learning Architecture for Music Emotion

Hortense Fong, Vineet Kumar, K. Sudhir

TL;DR

This study tackles predicting time-varying music emotion with an explainable deep learning approach that embeds harmonic theory into non-contiguous CNN filters. By using harmonics-based mel blinders and Grad-CAM explanations, the model associates consonance with positive and low-arousal states, achieving comparable predictive accuracy to atheoretical models but with far fewer parameters. The paper demonstrates practical value through an emotion-aware ad-insertion application, showing that emotionally congruent placements reduce skip rates and improve recall, while maintaining explainability. These results advance scalable, theory-grounded emotion prediction for music in marketing contexts and beyond, with potential extensions to multimodal data and longer temporal dependencies.

Abstract

This paper paper develops a theory-based, explainable deep learning convolutional neural network (CNN) classifier to predict the time-varying emotional response to music. We design novel CNN filters that leverage the frequency harmonics structure from acoustic physics known to impact the perception of musical features. Our theory-based model is more parsimonious, but provides comparable predictive performance to atheoretical deep learning models, while performing better than models using handcrafted features. Our model can be complemented with handcrafted features, but the performance improvement is marginal. Importantly, the harmonics-based structure placed on the CNN filters provides better explainability for how the model predicts emotional response (valence and arousal), because emotion is closely related to consonance--a perceptual feature defined by the alignment of harmonics. Finally, we illustrate the utility of our model with an application involving digital advertising. Motivated by YouTube mid-roll ads, we conduct a lab experiment in which we exogenously insert ads at different times within videos. We find that ads placed in emotionally similar contexts increase ad engagement (lower skip rates, higher brand recall rates). Ad insertion based on emotional similarity metrics predicted by our theory-based, explainable model produces comparable or better engagement relative to atheoretical models.

A Theory-Based Explainable Deep Learning Architecture for Music Emotion

TL;DR

This study tackles predicting time-varying music emotion with an explainable deep learning approach that embeds harmonic theory into non-contiguous CNN filters. By using harmonics-based mel blinders and Grad-CAM explanations, the model associates consonance with positive and low-arousal states, achieving comparable predictive accuracy to atheoretical models but with far fewer parameters. The paper demonstrates practical value through an emotion-aware ad-insertion application, showing that emotionally congruent placements reduce skip rates and improve recall, while maintaining explainability. These results advance scalable, theory-grounded emotion prediction for music in marketing contexts and beyond, with potential extensions to multimodal data and longer temporal dependencies.

Abstract

This paper paper develops a theory-based, explainable deep learning convolutional neural network (CNN) classifier to predict the time-varying emotional response to music. We design novel CNN filters that leverage the frequency harmonics structure from acoustic physics known to impact the perception of musical features. Our theory-based model is more parsimonious, but provides comparable predictive performance to atheoretical deep learning models, while performing better than models using handcrafted features. Our model can be complemented with handcrafted features, but the performance improvement is marginal. Importantly, the harmonics-based structure placed on the CNN filters provides better explainability for how the model predicts emotional response (valence and arousal), because emotion is closely related to consonance--a perceptual feature defined by the alignment of harmonics. Finally, we illustrate the utility of our model with an application involving digital advertising. Motivated by YouTube mid-roll ads, we conduct a lab experiment in which we exogenously insert ads at different times within videos. We find that ads placed in emotionally similar contexts increase ad engagement (lower skip rates, higher brand recall rates). Ad insertion based on emotional similarity metrics predicted by our theory-based, explainable model produces comparable or better engagement relative to atheoretical models.
Paper Structure (53 sections, 9 equations, 21 figures, 14 tables)

This paper contains 53 sections, 9 equations, 21 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Listener Response to Music
  4. Machine Learning with Unstructured Audio Data
  5. Ad Insertion in Videos
  6. Deep Learning Model for Music Emotion
  7. Model Elements
  8. (S1) Physical Properties of Sound Waves:
  9. (S2) Short-Time Fourier Transform Spectrogram:
  10. (S3) Mel Spectrogram based on Auditory Perception:
  11. (S4) Incorporating Harmonics-based Structure into CNN Filter Design:
  12. Harmonics.
  13. Pitch Class.
  14. Harmonics Filter Design.
  15. (S5) Convolutional Neural Network Architecture:
  16. ...and 38 more sections

Figures (21)

  • Figure 1: Music Emotion Classification Schematic
  • Figure 2: Example of Waveform and Spectrum
  • Figure 3: Waveform
  • Figure 4: Spectrum
  • Figure 5: Spectrograms
  • ...and 16 more figures