Aligning Audio Captions with Human Preferences

TL;DR

This work proposes a preference-aligned audio captioning framework based on Reinforcement Learning from Human Feedback (RLHF), which achieves performance comparable to supervised approaches with ground-truth data, demonstrating effective alignment with human preferences and scalability in real-world use.

Abstract

Current audio captioning relies on supervised learning with paired audio-caption data, which is costly to curate and may not reflect human preferences in real-world scenarios. To address this, we propose a preference-aligned audio captioning framework based on Reinforcement Learning from Human Feedback (RLHF). To capture nuanced preferences, we train a Contrastive Language-Audio Pretraining (CLAP) based reward model using human-labeled pairwise preference data. This reward model is integrated into an RL framework to fine-tune any baseline captioning system without ground-truth annotations. Extensive human evaluations across multiple datasets show that our method produces captions preferred over baseline models, particularly when baselines fail to provide correct and natural captions. Furthermore, our framework achieves performance comparable to supervised approaches with ground-truth data, demonstrating effective alignment with human preferences and scalability in real-world use.

Figures (2)

• Figure 1: Pairwise human preference annotation and subjective evaluation setup
• Figure 2: Proposed method for RL using custom reward model (left) and CLAP based architecture of the custom reward model inference(right).