Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interface Design for Self-Supervised Speech Models

Yi-Jen Shih, David Harwath

TL;DR

This work addresses how to best aggregate layer-wise representations from self-supervised speech models for downstream tasks by introducing an explicit Interface module between Upstream SSL models and Downstream predictors. It formalizes the Upstream→Interface→Downstream pipeline and shows that the conventional layer-wise weighted sum is suboptimal. Among several proposed interfaces, Hierarchical Convolution over the layer axis consistently delivers the strongest performance across ML-SUPERB and SUPERB benchmarks, often outperforming both single-layer features and larger downstream heads. The findings emphasize that interface design significantly impacts downstream performance, persisting even under end-to-end fine-tuning, and offer practical guidance for deploying SSL-based speech systems.

Abstract

Self-supervised speech (SSL) models have recently become widely adopted for many downstream speech processing tasks. The general usage pattern is to employ SSL models as feature extractors, and then train a downstream prediction head to solve a specific task. However, different layers of SSL models have been shown to capture different types of information, and the methods of combining them are not well studied. To this end, we extend the general framework for SSL model utilization by proposing the interface that connects the upstream and downstream. Under this view, the dominant technique of combining features via a layerwise weighted sum can be regarded as a specific interface. We propose several alternative interface designs and demonstrate that the weighted sum interface is suboptimal for many tasks. In particular, we show that a convolutional interface whose depth scales logarithmically with the depth of the upstream model consistently outperforms many other interface designs.

Interface Design for Self-Supervised Speech Models

TL;DR

This work addresses how to best aggregate layer-wise representations from self-supervised speech models for downstream tasks by introducing an explicit Interface module between Upstream SSL models and Downstream predictors. It formalizes the Upstream→Interface→Downstream pipeline and shows that the conventional layer-wise weighted sum is suboptimal. Among several proposed interfaces, Hierarchical Convolution over the layer axis consistently delivers the strongest performance across ML-SUPERB and SUPERB benchmarks, often outperforming both single-layer features and larger downstream heads. The findings emphasize that interface design significantly impacts downstream performance, persisting even under end-to-end fine-tuning, and offer practical guidance for deploying SSL-based speech systems.

Abstract

Self-supervised speech (SSL) models have recently become widely adopted for many downstream speech processing tasks. The general usage pattern is to employ SSL models as feature extractors, and then train a downstream prediction head to solve a specific task. However, different layers of SSL models have been shown to capture different types of information, and the methods of combining them are not well studied. To this end, we extend the general framework for SSL model utilization by proposing the interface that connects the upstream and downstream. Under this view, the dominant technique of combining features via a layerwise weighted sum can be regarded as a specific interface. We propose several alternative interface designs and demonstrate that the weighted sum interface is suboptimal for many tasks. In particular, we show that a convolutional interface whose depth scales logarithmically with the depth of the upstream model consistently outperforms many other interface designs.
Paper Structure (17 sections, 1 equation, 2 figures, 4 tables)

This paper contains 17 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Interface Methods
  3. Interface Definition
  4. Proposed Interfaces
  5. Grouped Weighted Sums
  6. Concatenation + Learnable Projection
  7. Hierarchical Convolution over Layers
  8. CLS Pooling over layer dimension
  9. PCA + Concatenation
  10. Experiments
  11. Experimental Setup
  12. Interface Comparison on ML-SUPERB Monolingual ASR with HuBERT Base
  13. Interface Comparison on ML-SUPERB and SUPERB
  14. Interface vs. Larger Downstream Models
  15. Performance under Fine-tuning setting
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The general framework for utilizing self-supervised speech models only considers the upstream and downstream model components. We argue that the interface connecting them should be considered in its own right as a separate component.($L$ : number of upstream model layers, $T$ : upstream model output sequence length, $D$ : upstream model feature dimension)
  • Figure 2: Proposed Interface designs. (a): Hierarchical Convolution is applied over the layer dimension of the upstream model hiddenstates. (b): A learnable CLS embedding is used for summarizing over layer dimension