What Do Language Models Hear? Probing for Auditory Representations in Language Models

TL;DR

The paper investigates whether text-only language models encode perceptual auditory representations by introducing a contrastive probing framework that learns linear projections to align LM text embeddings with audio embeddings from pretrained models. The probe, evaluated with a loss $L_C = \sum_{c \in \mathcal{C}} \left( - s(text(c), sound(c)) / \tau + \log \left( \sum_{c' \in N(c)} \exp( s(text(c'), sound(c)) / \tau ) \right) \right)$, is trained on a train set and tested zero-shot on unseen classes using accuracy@K. Across six language models and three audio models on FSD50K, the probe achieves above-chance generalization, with supervised audio representations and larger models providing stronger alignment. These findings suggest that grounded auditory knowledge can emerge from text-only training and underscore the importance of the quality and priors of audio representations for cross-modal grounding applications.

Abstract

This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, the probe is tested on its ability to generalize to objects that were not seen during training. Across different language models and audio models, we find that the probe generalization is above chance in many cases, indicating that despite being trained only on raw text, language models encode grounded knowledge of sounds for some objects.

Figures (7)

• Figure 1: (Top) Language (triangle) and sound (circle) representations aligned via Procrustes analysis schonemann_generalized_1966, visualized via PCA. The language representation is from BERT devlin-etal-2019-bert and the audio representation is from PaSST passt. The classes are color-coded based on their parent nodes (i.e., human voice, domestic sounds, animal, music) according to the ontology from the FSD50K fonseca2021fsd50k. (Bottom) A zoomed-in portion of the blue region of the top figure, which shows the structural similarities between the language and sound representations for the music category.
• Figure 2: An overview of our experimental setup. We randomly split a set of classes into mutually exclusive train/test sets. On the training set (blue), we use a contrastive loss to learn linear transformations (i.e., projection heads) of the sound and language representations such that a language representation of a class is close in cosine distance to the sound representation of the same class. After training, we apply the learned probe on audio snippets of classes from the test set, and retrieve the most similar text representation (from classes in both the train and test sets). We then test whether the retrieved class corresponds to the actual class.
• Figure 3: Accuracy@3 for the different language and sound representations. Green bars show the accuracy of the permuted embedding control task, where the text representations are randomly permuted. Error bars show standard error of the mean across 5 runs. Dotted line shows random chance performance, which is 2.08%.
• Figure 4: Classes that had the best accuracies (as measured by accuracy@3) for the different sound representations. We measure the accuracies across all 5 train/test sets, and average across the different language models.
• Figure 5: Rank correlation of accuracies of classes within the test set between language representations, where the correlations are averaged across the five runs.