ASK: Adaptive Self-improving Knowledge Framework for Audio Text Retrieval

TL;DR

<3-5 sentence high-level summary> The paper identifies two fundamental challenges in knowledge-enhanced Audio-Text Retrieval: the Gradient Locality Bottleneck (GLB), which confines learning to in-batch negatives, and the Representation Drift Mismatch (RDM), which arises when static external knowledge becomes misaligned with an evolving model. It proposes Adaptive Self-improving Knowledge (ASK), a model-agnostic framework that injects multi-grained knowledge, uses adaptive reliability weighting, and dynamically refines knowledge bases to co-evolve with the model. The authors formalize GLB and RDM, provide convergence analyses, and demonstrate state-of-the-art results on AudioCaps and Clotho across global and local interaction regimes, with extensive ablations confirming the importance of each component. ASK offers a general, plug-and-play approach to robust, knowledge-enhanced cross-modal retrieval and has implications for broader cross-modal learning where external knowledge must co-evolve with models.

Abstract

The dominant paradigm for Audio-Text Retrieval (ATR) relies on mini-batch-based contrastive learning. This process, however, is inherently limited by what we formalize as the Gradient Locality Bottleneck (GLB), which structurally prevents models from leveraging out-of-batch knowledge and thus impairs fine-grained and long-tail learning. While external knowledge-enhanced methods can alleviate the GLB, we identify a critical, unaddressed side effect: the Representation-Drift Mismatch (RDM), where a static knowledge base becomes progressively misaligned with the evolving model, turning guidance into noise. To address this dual challenge, we propose the Adaptive Self-improving Knowledge (ASK) framework, a model-agnostic, plug-and-play solution. ASK breaks the GLB via multi-grained knowledge injection, systematically mitigates RDM through dynamic knowledge refinement, and introduces a novel adaptive reliability weighting scheme to ensure consistent knowledge contributes to optimization. Experimental results on two benchmark datasets with superior, state-of-the-art performance justify the efficacy of our proposed ASK framework.

Figures (4)

• Figure 1: Comparison between the conventional batch-only paradigm (left) and our proposed ASK framework (right).
• Figure 2: The proposed ASK framework. A multi-grained knowledge base ($K_f, K_c$) is periodically updated to mitigate RDM. During training, knowledge is injected into samples ($u_i \to u'_i$), and a cross-modal reliability weight ($\Psi$) is computed. A final loss is optimized using both an OT-realigned similarity matrix ($S^*$) and the reliability weight $\Psi$.
• Figure 3: Ablation experiment on ASK$^+$. Effect of the number $\mathcal{T}$ of Knowledge Update.
• Figure 4: t-SNE visualization of Representation Drift. Embeddings of a fixed set of audio samples, encoded by the same model at different training epochs, are plotted. The progressive shift in embedding positions (from Epoch 1 [blue] to Epoch 50 [red]) empirically validates the core premise of RDM: a static knowledge base becomes misaligned with the non-stationary representation space over time.