Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimating Musical Surprisal in Audio

Mathias Rose Bjare, Giorgia Cantisani, Stefan Lattner, Gerhard Widmer

TL;DR

This work extends information-content (IC) based surprisal estimation from symbolic music to full-length audio by training a Transformer to predict 64-dimensional Music2Latent representations and modeling the next-frame likelihood with a 32-component Gaussian Mixture Model. IC is shown to decrease with repetition, higher for later segments, and related to timbral changes and loudness, with significant correlations to complexity measures and the ability to predict EEG responses to songs. The approach offers a general audio-domain perceptual surprisal model and demonstrates practical applicability with open-source code. This bridges symbolic and audio analyses, enabling robust, data-driven investigations of musical expectation in real-world audio.

Abstract

In modeling musical surprisal expectancy with computational methods, it has been proposed to use the information content (IC) of one-step predictions from an autoregressive model as a proxy for surprisal in symbolic music. With an appropriately chosen model, the IC of musical events has been shown to correlate with human perception of surprise and complexity aspects, including tonal and rhythmic complexity. This work investigates whether an analogous methodology can be applied to music audio. We train an autoregressive Transformer model to predict compressed latent audio representations of a pretrained autoencoder network. We verify learning effects by estimating the decrease in IC with repetitions. We investigate the mean IC of musical segment types (e.g., A or B) and find that segment types appearing later in a piece have a higher IC than earlier ones on average. We investigate the IC's relation to audio and musical features and find it correlated with timbral variations and loudness and, to a lesser extent, dissonance, rhythmic complexity, and onset density related to audio and musical features. Finally, we investigate if the IC can predict EEG responses to songs and thus model humans' surprisal in music. We provide code for our method on github.com/sonycslparis/audioic.

Estimating Musical Surprisal in Audio

TL;DR

This work extends information-content (IC) based surprisal estimation from symbolic music to full-length audio by training a Transformer to predict 64-dimensional Music2Latent representations and modeling the next-frame likelihood with a 32-component Gaussian Mixture Model. IC is shown to decrease with repetition, higher for later segments, and related to timbral changes and loudness, with significant correlations to complexity measures and the ability to predict EEG responses to songs. The approach offers a general audio-domain perceptual surprisal model and demonstrates practical applicability with open-source code. This bridges symbolic and audio analyses, enabling robust, data-driven investigations of musical expectation in real-world audio.

Abstract

In modeling musical surprisal expectancy with computational methods, it has been proposed to use the information content (IC) of one-step predictions from an autoregressive model as a proxy for surprisal in symbolic music. With an appropriately chosen model, the IC of musical events has been shown to correlate with human perception of surprise and complexity aspects, including tonal and rhythmic complexity. This work investigates whether an analogous methodology can be applied to music audio. We train an autoregressive Transformer model to predict compressed latent audio representations of a pretrained autoencoder network. We verify learning effects by estimating the decrease in IC with repetitions. We investigate the mean IC of musical segment types (e.g., A or B) and find that segment types appearing later in a piece have a higher IC than earlier ones on average. We investigate the IC's relation to audio and musical features and find it correlated with timbral variations and loudness and, to a lesser extent, dissonance, rhythmic complexity, and onset density related to audio and musical features. Finally, we investigate if the IC can predict EEG responses to songs and thus model humans' surprisal in music. We provide code for our method on github.com/sonycslparis/audioic.
Paper Structure (14 sections, 2 equations, 2 figures, 1 table)

This paper contains 14 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Audio Encoding
  5. Modelling Suprisal
  6. Training Details
  7. Data
  8. Model and Training
  9. Experiments
  10. Surprisal Reduction with Repetition
  11. Contrasting Segments Hypothesis
  12. Relation to Complexity
  13. Prediction of EEG Responses
  14. Conclusion

Figures (2)

  • Figure 1: Temporal development of correlations between IC and dissonance $(d)$, IOI-entropy $(r)$, onset density $(o)$ loudness $(l)$ and spectral flux $(f)$ on the dataset PR.
  • Figure 2: z-transformed Pearson's correlation gain for individual channels on scalp topographies (FDR corrected t-test with significance threshold set at $p<0.05$) and for the average over all channels (bar plot, mean+-SE across participants and trials, red dots represent individual participants).