Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

Francesco Paissan, Elisabetta Farella

TL;DR

This paper investigates how to reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that is called tinyCLAP, which uses only 6% of the original Microsoft CLAP parameters with a minimal reduction in zero-shot classification performance.

Abstract

Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

TL;DR

This paper investigates how to reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that is called tinyCLAP, which uses only 6% of the original Microsoft CLAP parameters with a minimal reduction in zero-shot classification performance.

Abstract

Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested
Paper Structure (13 sections, 9 equations, 2 figures, 2 tables)

This paper contains 13 sections, 9 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Contrastive Language-Audio Pretraining
  4. Distilling Audio Representations without Text
  5. Pruning the Shared Multimodal Latent Space
  6. Efficient Audio Encoder
  7. Experimental setup
  8. Results
  9. Distillation Sanity Checks
  10. Reducing Computational Complexity
  11. Conclusions
  12. Acknowledgements
  13. Results on Entire Evaluation Setup

Figures (2)

  • Figure 1: Diagram of the proposed distillation technique. During distillation, the student and audio encoders are aligned by minimizing the loss in Eq. \ref{['eq:loss']}. During the distillation stage, the encoders represented in grey are frozen, while those in green are trained. For simplicity, the image does not include the projection layers.
  • Figure 2: a) Impact of the latent space pruning on the distilled checkpoints. The pruning procedure is described in Sec \ref{['sec:pruning']}. b) Computational complexity against ZS performance of the distilled models. Note that the performance of the distilled and the original models match, considering the baseline distillation scenario. Similar plots, computed for all datasets independently, can be found on the companion website.