Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha

TL;DR

This paper proposes ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild, and proposes prompt augmentation, a traditional method of employing hand-written template prompts that generate custom prompts for each unique label in the dataset.

Abstract

Open-vocabulary audio-language models, like CLAP, offer a promising approach for zero-shot audio classification (ZSAC) by enabling classification with any arbitrary set of categories specified with natural language prompts. In this paper, we propose a simple but effective method to improve ZSAC with CLAP. Specifically, we shift from the conventional method of using prompts with abstract category labels (e.g., Sound of an organ) to prompts that describe sounds using their inherent descriptive features in a diverse context (e.g.,The organ's deep and resonant tones filled the cathedral.). To achieve this, we first propose ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild. These rewritten captions describe each sound event in the original caption using their unique discriminative characteristics. ReCLAP outperforms all baselines on both multi-modal audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification with ReCLAP, we propose prompt augmentation. In contrast to the traditional method of employing hand-written template prompts, we generate custom prompts for each unique label in the dataset. These custom prompts first describe the sound event in the label and then employ them in diverse scenes. Our proposed method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all baselines by 1% - 55%.

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

TL;DR

This paper proposes ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild, and proposes prompt augmentation, a traditional method of employing hand-written template prompts that generate custom prompts for each unique label in the dataset.

Abstract

Open-vocabulary audio-language models, like CLAP, offer a promising approach for zero-shot audio classification (ZSAC) by enabling classification with any arbitrary set of categories specified with natural language prompts. In this paper, we propose a simple but effective method to improve ZSAC with CLAP. Specifically, we shift from the conventional method of using prompts with abstract category labels (e.g., Sound of an organ) to prompts that describe sounds using their inherent descriptive features in a diverse context (e.g.,The organ's deep and resonant tones filled the cathedral.). To achieve this, we first propose ReCLAP, a CLAP model trained with rewritten audio captions for improved understanding of sounds in the wild. These rewritten captions describe each sound event in the original caption using their unique discriminative characteristics. ReCLAP outperforms all baselines on both multi-modal audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification with ReCLAP, we propose prompt augmentation. In contrast to the traditional method of employing hand-written template prompts, we generate custom prompts for each unique label in the dataset. These custom prompts first describe the sound event in the label and then employ them in diverse scenes. Our proposed method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all baselines by 1% - 55%.
Paper Structure (11 sections, 5 equations, 2 figures, 5 tables)

This paper contains 11 sections, 5 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Caption Augmentation for ReCLAP
  5. Prompt Augmentation for ZSAC
  6. Experimental Setup
  7. Results
  8. Result Analysis
  9. Hyper-parameter Tuning
  10. Conclusion
  11. Limitations and Future Work

Figures (2)

  • Figure 1: Illustration of our proposed method for improving Zero Shot Audio Classification (ZSAC) with language augmentation. Top: We enhance CLAP training through caption augmentation, where each audio's caption is expanded and rewritten by prompting LLMs to provide detailed descriptions of the sound events. During training, we choose either the original caption or one of the rewritten captions. Bottom: We perform prompt augmentation and generate custom prompts for each label category in the dataset. These prompts describe the sound in the category in diverse scenes.
  • Figure 2: Comparison of accurately classified instances for 4 labels.