A Theoretical Framework for Acoustic Neighbor Embeddings

TL;DR

The paper tackles the challenge of interpreting acoustic neighbor embeddings by proposing a probabilistic framework where distances between audio and text embeddings reflect phonetic similarity. It formalizes a fundamental definition of similarity via overlap integrals and Bayes error, and derives tractable approximations under Gaussian, isotropic assumptions that reduce to simple Euclidean distances. A complete training pipeline is described, with audio and text embedders aligned through a binary-distance objective and mean-squared-error alignment, and empirical evidence is provided across four diverse experiments that validate the framework and demonstrate practical applications such as word classification, OOV recovery, dialect clustering, and wake-word prediction. The results suggest that, under a cluster-wise isotropy approximation, acoustic-neighbor distances can be interpreted as Gaussian likelihoods and overlap-based similarities, enabling principled, scalable use of embeddings in real-world speech and text tasks, with code and models publicly available for replication.

Abstract

This paper provides a theoretical framework for interpreting acoustic neighbor embeddings, which are representations of the phonetic content of variable-width audio or text in a fixed-dimensional embedding space. A probabilistic interpretation of the distances between embeddings is proposed, based on a general quantitative definition of phonetic similarity between words. This provides us a framework for understanding and applying the embeddings in a principled manner. Theoretical and empirical evidence to support an approximation of uniform cluster-wise isotropy are shown, which allows us to reduce the distances to simple Euclidean distances. Four experiments that validate the framework and demonstrate how it can be applied to diverse problems are described. Nearest-neighbor search between audio and text embeddings can give isolated word classification accuracy that is identical to that of finite state transducers (FSTs) for vocabularies as large as 500k. Embedding distances give accuracy with 0.5% point difference compared to phone edit distances in out-of-vocabulary word recovery, as well as producing clustering hierarchies identical to those derived from human listening experiments in English dialect clustering. The theoretical framework also allows us to use the embeddings to predict the expected confusion of device wake-up words. All source code and pretrained models are provided.

Figures (11)

• Figure 1: A definition of phonetic similarity between two words via a hypothetical physical experiment that is abstracted in probabilistic terms
• Figure 2: Example 2-dimensional embedding space where an audio embedder and text embedder are trained over diverse audio samples for the words "fifty", "fifteen", and "thirteen."
• Figure 3: Fig. \ref{['fig:g-training']} where we now approximate each cluster to have a Gaussian distribution with equal isotropic covariances represented by the dotted circles.
• Figure 4: Normalized distribution of word pronunciation lengths (i.e., number of phones) used for encoder training.
• Figure 5: Hypothesized distribution shift for two different dialects of English for a given word $w_c$. If the embedding centroid for $w_c$ over general speakers is ${\mathbf g}_c$, we hypothesize that each dialect manifests as a shift of the general distribution in different directions, resulting in centroids ${\mathbf g}_c^{(1)}$ and ${\mathbf g}_c^{(2)}$ for Dialects 1 and 2, respectively. The (shaded) area of overlap between the two shifted distributions is the overall phonetic similarity between the two dialects' examples of word $w_c$.