EDTC: enhance depth of text comprehension in automated audio captioning

TL;DR

This work tackles the persistent modality gap between audio and text in Automated Audio Captioning by introducing Enhance Depth of Text Comprehension (EDTC), which combines three mechanisms: FUSER for multi-encoder feature fusion, TRANSLATOR for tensor-level alignment, and a twin-structure with momentum updates to learn both modalities concurrently. The audio branch leverages three frozen encoders, while the text branch uses a frozen INSTRUCTOR encoder; a trainable BART decoder generates captions, guided by a cross-entropy loss and a CLIP-inspired contrastive loss to align modalities. Key contributions include a plug-and-play FUSER for richer semantic fusion, a tensor-level TRANSLATOR for robust cross-modal alignment, and a momentum-updated twin TRANSLATOR design that enables simultaneous cross-modal learning without heavy reliance on reinforcement learning. Empirically, EDTC achieves state-of-the-art results on AudioCaps and competitive performance on Clotho, with ablations confirming the effectiveness of each component and suggesting strong practical impact for multimodal captioning systems.

Abstract

Modality discrepancies have perpetually posed significant challenges within the realm of Automated Audio Captioning (AAC) and across all multi-modal domains. Facilitating models in comprehending text information plays a pivotal role in establishing a seamless connection between the two modalities of text and audio. While recent research has focused on closing the gap between these two modalities through contrastive learning, it is challenging to bridge the difference between both modalities using only simple contrastive loss. This paper introduces Enhance Depth of Text Comprehension (EDTC), which enhances the model's understanding of text information from three different perspectives. First, we propose a novel fusion module, FUSER, which aims to extract shared semantic information from different audio features through feature fusion. We then introduced TRANSLATOR, a novel alignment module designed to align audio features and text features along the tensor level. Finally, the weights are updated by adding momentum to the twin structure so that the model can learn information about both modalities at the same time. The resulting method achieves state-of-the-art performance on AudioCaps datasets and demonstrates results comparable to the state-of-the-art on Clotho datasets.

Figures (5)

• Figure 1: Overview of the methodology presented in this paper. Audio encoder uses PANNskong2020panns, HTSATchen2022hts, ConvNeXtliu2022convnxet for extracting audio features. INSTRUCTOR-basesu2022Instructior is used as Text Encoder for extracting text features and BART is used as Decoder for generating captions. All encoder weights are frozen.
• Figure 2: The statement about the reason for selecting FUSER, where the orange dialog box represents the audio sample, and the blue dialog box represents the text corresponding to each piece of audio.
• Figure 3: Detail diagram of the FUSER module. The module consists of an Input layer for the pre-fusion and a Fusion layer for the L layers. The input of the module is the audio features extracted for each encoder, and the output is the final fusion features.
• Figure 4: The statement about the reason for selecting TRANSLATOR, where time_step is the time step, n_mel is the number of mel filters, seq_len is the text length, d_model denotes the encoding dimension of each word.
• Figure 5: Detail diagram of the TRANSLATOR module, which is composed of an M-layer Encoder Layer and an N-layer Decoder Layer. The module input is split into individual time steps, and then the output of each time step is concatenated into a complete output after passing through the module.