Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training

TL;DR

This work tackles the challenge of fine-grained speaking-style modeling in speech–text pre-training by introducing FCaps, a large-scale dataset of 47k hours with 19M fine-grained captions grounded end-to-end in audio. Building on FCaps, the authors propose CLSP, a dual-encoder model trained with a two-stage curriculum that jointly leverages global and fine-grained captions to learn multi-granular representations. Across global and fine-grained speech–text retrieval, zero-shot paralinguistic classification, and human-aligned speech style similarity tasks, CLSP outperforms strong baselines and aligns well with human judgments. The work demonstrates the practicality and impact of open-vocabulary, cross-granular, audio-grounded descriptions for robust speech–language representations and provides publicly available resources for future research.

Abstract

Modeling fine-grained speaking styles remains challenging for language-speech representation pre-training, as existing speech-text models are typically trained with coarse captions or task-specific supervision, and scalable fine-grained style annotations are unavailable. We present FCaps, a large-scale dataset with fine-grained free-text style descriptions, encompassing 47k hours of speech and 19M fine-grained captions annotated via a novel end-to-end pipeline that directly grounds detailed captions in audio, thereby avoiding the error propagation caused by LLM-based rewriting in existing cascaded pipelines. Evaluations using LLM-as-a-judge demonstrate that our annotations surpass existing cascaded annotations in terms of correctness, coverage, and naturalness. Building on FCaps, we propose CLSP, a contrastive language-speech pre-trained model that integrates global and fine-grained supervision, enabling unified representations across multiple granularities. Extensive experiments demonstrate that CLSP learns fine-grained and multi-granular speech-text representations that perform reliably across global and fine-grained speech-text retrieval, zero-shot paralinguistic classification, and speech style similarity scoring, with strong alignment to human judgments. All resources will be made publicly available.

Figures (9)

• Figure 1: Multi-granular speech style caption similarity scoring for the same speech input by CLSP. Positive captions (green) receive higher scores, while hard negatives (red), despite mainly textual overlap, receive markedly lower scores due to attribute mismatches.
• Figure 2: Overview of our end-to-end annotation pipeline for generating fine-grained captions, consisting of a detailed captioner and agentic verification with specialist tools. Solid lines indicate the construction process for FCaps-Emilia, and dashed lines indicate additional processes for FCaps-PSCBase. In the example fine-grained captions, speaker-related traits are highlighted in bold and narrative structure in red.
• Figure 3: Overview of CLSP.
• Figure 4: Pairwise comparison between end-to-end and cascaded captions across correctness, coverage, and naturalness dimensions, showing the proportions of better, tied, and worse cases under Gemini 3 Pro evaluation.
• Figure 5: Annotation UI for raters to annotate the alignment score between one audio and several candidate captions.