Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs

Wenxi Chen, Ziyang Ma, Xiquan Li, Xuenan Xu, Yuzhe Liang, Zhisheng Zheng, Kai Yu, Xie Chen

TL;DR

This paper proposes SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs, which achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.

Abstract

Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible. In this paper, we propose SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs. Our approach uses the self-supervised EAT model to extract fine-grained audio representations, which are then aligned with textual embeddings via lightweight linear layers. The caption generation LLM is efficiently fine-tuned using the LoRA adapter. Drawing inspiration from the back-translation method in machine translation, we implement paraphrasing augmentation to expand the Clotho dataset during pre-training. This strategy helps alleviate the limitation of scarce audio-text pairs and generates more diverse captions from a small set of audio clips. During inference, we introduce the plug-and-play CLAP-Refine strategy to fully exploit multiple decoding outputs, akin to the n-best rescoring strategy in speech recognition. Using the CLAP model for audio-text similarity calculation, we could select the textual descriptions generated by multiple searching beams that best match the input audio. Experimental results show that SLAM-AAC achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.

SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs

TL;DR

This paper proposes SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs, which achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.

Abstract

Automated Audio Captioning (AAC) aims to generate natural textual descriptions for input audio signals. Recent progress in audio pre-trained models and large language models (LLMs) has significantly enhanced audio understanding and textual reasoning capabilities, making improvements in AAC possible. In this paper, we propose SLAM-AAC to further enhance AAC with paraphrasing augmentation and CLAP-Refine through LLMs. Our approach uses the self-supervised EAT model to extract fine-grained audio representations, which are then aligned with textual embeddings via lightweight linear layers. The caption generation LLM is efficiently fine-tuned using the LoRA adapter. Drawing inspiration from the back-translation method in machine translation, we implement paraphrasing augmentation to expand the Clotho dataset during pre-training. This strategy helps alleviate the limitation of scarce audio-text pairs and generates more diverse captions from a small set of audio clips. During inference, we introduce the plug-and-play CLAP-Refine strategy to fully exploit multiple decoding outputs, akin to the n-best rescoring strategy in speech recognition. Using the CLAP model for audio-text similarity calculation, we could select the textual descriptions generated by multiple searching beams that best match the input audio. Experimental results show that SLAM-AAC achieves state-of-the-art performance on Clotho V2 and AudioCaps, surpassing previous mainstream models.

Paper Structure

This paper contains 13 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. SLAM-AAC
  3. Network Architecture
  4. Paraphrasing Augmentation in AAC
  5. CLAP-Refine for Text Decoding
  6. Experimental Setup
  7. Datasets
  8. Evaluation Metrics
  9. Training and Inference Details
  10. Experimental Results
  11. Main Results
  12. Ablation Study
  13. Conclusion and Future Work

Figures (1)

  • Figure 1: Overview of the SLAM-AAC system. We use the frozen EAT to extract fine-grained audio representations, which are then downsampled and aligned with text embeddings via a linear projector. The LLM for decoding generates text based on these concatenated representations and is efficiently fine-tuned using LoRA. During inference, multiple candidate captions are generated through various beam searches, with the most audio-aligned textual description selected as the final output using the CLAP-Refine strategy. Here, $B_n$ denotes the candidate generated using a beam size of $n$ in decoding.