Learning Audio Concepts from Counterfactual Natural Language

TL;DR

The paper addresses learning audio representations beyond fixed class labels by introducing counterfactual natural language to guide audio-text pretraining. It prompts an LLM to identify acoustic sources and generate counterfactual captions, integrating this into a CLAP-like framework with a dual loss that enforces factual consistency while distinguishing counterfactuals. Pretraining on AudioCaps, Clotho, and MACS with frozen encoders yields significant gains in open-ended text-based audio retrieval (notably a top-1 increase of about 43%), with mixed results in zero-shot benchmarks across ESC-50 and US8K. The work demonstrates the feasibility and impact of counterfactual reasoning in audio and points to future work on deeper causal levels and broader datasets.

Abstract

Conventional audio classification relied on predefined classes, lacking the ability to learn from free-form text. Recent methods unlock learning joint audio-text embeddings from raw audio-text pairs describing audio in natural language. Despite recent advancements, there is little exploration of systematic methods to train models for recognizing sound events and sources in alternative scenarios, such as distinguishing fireworks from gunshots at outdoor events in similar situations. This study introduces causal reasoning and counterfactual analysis in the audio domain. We use counterfactual instances and include them in our model across different aspects. Our model considers acoustic characteristics and sound source information from human-annotated reference texts. To validate the effectiveness of our model, we conducted pre-training utilizing multiple audio captioning datasets. We then evaluate with several common downstream tasks, demonstrating the merits of the proposed method as one of the first works leveraging counterfactual information in audio domain. Specifically, the top-1 accuracy in open-ended language-based audio retrieval task increased by more than 43%.

Figures (3)

• Figure 1: Counterfactual reasoning helps to distinguish various sound sources in an audio signal captured by a microphone. We identify isolated sound sources using GPT-3.5-Turbo. Subsequently, we intervene to alter one or more of the sources of sounds to construct an imaginative linguistic representation in an alternative world that could have happened if there were other objects instead, thereby eliminating dependence on empirical audio data to reason objects driving the acoustic waves.
• Figure 2: (a) The CLAP method clap_paper utilizes audio captions to train audio embeddings. (b) Our method leverages a prompt, represented by $p = \{ p_1, p_2 \}$, using the GPT-3.5-Turbo model as LLM, that elicits counterfactual captions. This model guides interventions on the existing captions. The overarching goal is to pinpoint the sources of acoustic waves through human-narrated captions, shown as identification. After identifying these sources, controlling modification and intervention are performed by $p_2$ to incorporate the resulting counterfactual scenarios into a causal learning framework. This technique significantly improves the capability of audio-text models to distinguish subtle variations in text to align with sounds emanating from different objects.
• Figure 3: t-SNE visualization of audio embeddings with ours and CLAP compared to original and counterfactual caption embeddings under different parameter configurations. Visualization keys: factual text (red dots), counterfactuals (blue), our audio embeddings (green), and CLAP audio embeddings (orange). As loss terms are incrementally introduced, our audio embeddings consistently align more closely with factual data and distance from counterfactuals. Our audio embeddings get closer to facts and distance from the counterfactuals for various combinations of angle loss and factual consistency loss terms.