Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision Language Models Are Few-Shot Audio Spectrogram Classifiers

Satvik Dixit, Laurie M. Heller, Chris Donahue

TL;DR

It is demonstrated that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images, and it is shown that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset.

Abstract

We demonstrate that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images. Specifically, we instruct VLMs to perform audio classification tasks in a few-shot setting by prompting them to classify a spectrogram image given example spectrogram images of each class. By carefully designing the spectrogram image representation and selecting good few-shot examples, we show that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset. Moreover, we demonstrate that VLMs currently outperform the only available commercial audio language model with audio understanding capabilities (Gemini-1.5) on the equivalent audio classification task (59.00% vs. 49.62%), and even perform slightly better than human experts on visual spectrogram classification (73.75% vs. 72.50% on first fold). We envision two potential use cases for these findings: (1) combining the spectrogram and language understanding capabilities of VLMs for audio caption augmentation, and (2) posing visual spectrogram classification as a challenge task for VLMs.

Vision Language Models Are Few-Shot Audio Spectrogram Classifiers

TL;DR

It is demonstrated that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images, and it is shown that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset.

Abstract

We demonstrate that vision language models (VLMs) are capable of recognizing the content in audio recordings when given corresponding spectrogram images. Specifically, we instruct VLMs to perform audio classification tasks in a few-shot setting by prompting them to classify a spectrogram image given example spectrogram images of each class. By carefully designing the spectrogram image representation and selecting good few-shot examples, we show that GPT-4o can achieve 59.00% cross-validated accuracy on the ESC-10 environmental sound classification dataset. Moreover, we demonstrate that VLMs currently outperform the only available commercial audio language model with audio understanding capabilities (Gemini-1.5) on the equivalent audio classification task (59.00% vs. 49.62%), and even perform slightly better than human experts on visual spectrogram classification (73.75% vs. 72.50% on first fold). We envision two potential use cases for these findings: (1) combining the spectrogram and language understanding capabilities of VLMs for audio caption augmentation, and (2) posing visual spectrogram classification as a challenge task for VLMs.

Paper Structure

This paper contains 14 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Tasks & Methods
  3. Task definition for visual spectrogram classification (VSC)
  4. Default spectrogram extraction hyperparameters
  5. Prompting VLMs to perform VSC
  6. Experiments & Results
  7. Dataset and Models
  8. Comparing zero-shot performance of VLMs on the visual spectrogram classification task
  9. Few-shot performance of VLMs on the visual spectrogram classification task
  10. Human expert evaluations
  11. Cross-validated task performance
  12. Limitations
  13. Conclusion
  14. Acknowledgements

Figures (6)

  • Figure 1: Experimental setup of the visual spectrogram classification task in the few-shot setting
  • Figure 2: Example audio spectrograms for each class in the ESC-10 dataset
  • Figure 3: Confusion matrices for GPT-4o (left) and the ensembled human expert predictions (right)
  • Figure 4: Example spectrograms for the same audio using different configurations described in the hyperparameter ablations section
  • Figure 5: The template for prompting the VLM for zero-shot VSC. {image} is replaced by the spectrogram image to be classified.
  • ...and 1 more figures