Aligned Contrastive Predictive Coding

TL;DR

ACPC addresses learning self-supervised speech representations with content-aligned timing by aligning a short horizon of predictions $K$ to a longer latent horizon $M$ using Dynamic Time Warping, instead of enforcing exact future timing as in CPC. The method uses a strided convolutional encoder, a two-layer LSTM context model, and a Transformer-based prediction head, with a GPU-CTC-based DTW approximation and a contrastive loss over $N$ negatives. Empirically, ACPC yields smoother, more phoneme-aligned latent representations, improves ABX error rates by a notable margin, and increases frame-wise linear phone classification accuracy, while also offering up to ~1.7x faster training due to fewer prediction heads. These results suggest ACPC as a step toward segmentation-aware, linguistically meaningful self-supervised speech representations that scale to large datasets and reduce reliance on exact temporal timing.

Abstract

We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned. In this way, the prediction network solves a simpler task of predicting the next symbols, but not their exact timing, while the encoding network is trained to produce piece-wise constant latent codes. We evaluate the model on a speech coding task and demonstrate that the proposed Aligned Contrastive Predictive Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX error rates, while being slightly faster to train due to the reduced number of prediction heads.

Figures (5)

• Figure 1: ACPC architecture. The encoder maps chunks of input data into a latent space and the autoregressive model predicts $K$ upcoming latent vectors. They are aligned using DTW to the $M$ upcoming latent vectors. Training relies on a contrastive loss: the match between a prediction and its aligned latent vectors has to be stronger than the match of the predictor to any other latent vector. Using $K\mkern1.5mu{=}\mkern1.5mu M$ is equivalent to CPC oord_representation_2018.
• Figure 2: Dot-product similarity of latent representations at the output of the convolutional encoder. The white vertical bars and tick labels indicate ground-truth phoneme segmentation. Moire-like artifacts are visible for CPC, but less pronounced for ACPC (see main text for details). The pronounced blocks on the diagonal align with human-annotated phonemes. Visually, ACPC yields a representation in which latent vectors that correspond to a single phone are more self-similar.
• Figure 3: Distribution of cosine similarities between representations of random pairs of frames (top) and consecutive frames (bottom). The histograms are computed on both the representations from the baseline CPC and those of the proposed ACPC.
• Figure 4: Clustering quality (solid lines, left axis) and agreement with the phone alignments (dashed lines, right axis) of the CPC and ACPC representations for different numbers of centroids. The quality is measured by the average squared distance to the assigned centroid, the agreement computed by Normalized Mutual Information (NMI) between the cluster assignments and ground truth phones, forced-aligned with ASR.
• Figure 5: Frame-wise linear phoneme prediction accuracy during training. ACPC steadily improves upon CPC, providing representations easier to classify.