SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set
William Havard, Laurent Besacier, Olivier Rosec
TL;DR
This work addresses the scarcity of large-scale visually grounded spoken corpora by augmenting MSCOCO with 616,767 synthetic spoken captions (≈600 hours) produced via concatenative TTS across eight voices, accompanied by word/syllable/phoneme timecodes. It introduces a rich metadata and tooling framework (SQLite, Praat TextGrid export) and demonstrates variability through speed perturbation and disfluencies, plus a preliminary unsupervised term discovery study to gauge segmentation challenges. The dataset also includes a Japanese translation augmentation to enable cross-language investigations, expanding potential LaVi tasks such as spoken-captioning, cross-modal retrieval, and grounded language learning. Overall, SPEECH-COCO provides a scalable resource for multimodal speech research and baseline analyses, with practical applications in speech-vision alignment and unsupervised linguistic unit discovery.
Abstract
This paper presents an augmentation of MSCOCO dataset where speech is added to image and text. Speech captions are generated using text-to-speech (TTS) synthesis resulting in 616,767 spoken captions (more than 600h) paired with images. Disfluencies and speed perturbation are added to the signal in order to sound more natural. Each speech signal (WAV) is paired with a JSON file containing exact timecode for each word/syllable/phoneme in the spoken caption. Such a corpus could be used for Language and Vision (LaVi) tasks including speech input or output instead of text. Investigating multimodal learning schemes for unsupervised speech pattern discovery is also possible with this corpus, as demonstrated by a preliminary study conducted on a subset of the corpus (10h, 10k spoken captions). The dataset is available on Zenodo: https://zenodo.org/record/4282267