Classifier-Guided Captioning Across Modalities

TL;DR

The paper addresses generalization gaps in captioning across modalities by recognizing that language models trained on image-caption data struggle to capture modality-specific semantics such as audibility. It introduces a modality-agnostic inference-time framework that couples a frozen LM with a binary audibility classifier to steer caption generation toward more audible descriptions without updating the LM’s weights, using a calibrated loss that combines $\mathcal{L}_{CE}$ and $\mathcal{L}_{classifier}$ with $\mathcal{L} = \lambda_0 \cdot \mathcal{L}_{CE} + \lambda_1 \cdot \mathcal{L}_{classifier}$ where $\lambda_0=0.2$ and $\lambda_1=0.6$. The audibility classifier is trained on a GPT-4–generated dataset and operates during inference to adjust the LM’s context cache $C_i$, enabling more audible captions in audio and other modalities. Evaluations on AudioCaps and Clotho show improved performance, including state-of-the-art results in zero-shot audio captioning, and ablations on image captioning demonstrate broad applicability. This approach offers practical cross-modal adaptability with limited training and demonstrates the value of inference-time guidance for aligning captions with domain-specific semantics.

Abstract

Most current captioning systems use language models trained on data from specific settings, such as image-based captioning via Amazon Mechanical Turk, limiting their ability to generalize to other modality distributions and contexts. This limitation hinders performance in tasks like audio or video captioning, where different semantic cues are needed. Addressing this challenge is crucial for creating more adaptable and versatile captioning frameworks applicable across diverse real-world contexts. In this work, we introduce a method to adapt captioning networks to the semantics of alternative settings, such as capturing audibility in audio captioning, where it is crucial to describe sounds and their sources. Our framework consists of two main components: (i) a frozen captioning system incorporating a language model (LM), and (ii) a text classifier that guides the captioning system. The classifier is trained on a dataset automatically generated by GPT-4, using tailored prompts specifically designed to enhance key aspects of the generated captions. Importantly, the framework operates solely during inference, eliminating the need for further training of the underlying captioning model. We evaluate the framework on various models and modalities, with a focus on audio captioning, and report promising results. Notably, when combined with an existing zero-shot audio captioning system, our framework improves its quality and sets state-of-the-art performance in zero-shot audio captioning.

Figures (1)

• Figure 1: An overview of our modality-agnostic approach is depicted, with a focus on audio captioning during inference. In this illustration, the audio modality is highlighted (enclosed in a green circle), where the captioner functions as an audio captioner, and the classifier is an audibility classifier (which was pre-trained on our audibility dataset, see Sec. \ref{['Audibility classifier']}). The model is guided to generate the phrase 'barking' instead of 'walking' by adjusting the context (C) using the gradients from the classifier loss $L_{classifier}$, as indicated by the red arrow. To maintain the inherent characteristics of the language model, the minimum divergence from the original distribution is optimized using the cross-entropy loss $L_{ce}$.