Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval

Andreea-Maria Oncescu, João F. Henriques, Andrew Zisserman, Samuel Albanie, A. Sophia Koepke

TL;DR

This work considers the egocentric video setting and proposes three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset, and confirms that LLMs can be used to determine the difficulty of identifying the action associated with a sound.

Abstract

Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio information from video-text datasets, we introduce a methodology for generating audio-centric descriptions using Large Language Models (LLMs). In this work, we consider the egocentric video setting and propose three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining audio-centric descriptions gives significantly higher zero-shot performance than using the original visual-centric descriptions. Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset. Finally, we confirm that LLMs can be used to determine the difficulty of identifying the action associated with a sound.

A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval

TL;DR

This work considers the egocentric video setting and proposes three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset, and confirms that LLMs can be used to determine the difficulty of identifying the action associated with a sound.

Abstract

Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, treating visual descriptions as audio descriptions is far from optimal. Even if audio class labels are present, they commonly are not very detailed, making them unsuited for text-audio retrieval. To exploit relevant audio information from video-text datasets, we introduce a methodology for generating audio-centric descriptions using Large Language Models (LLMs). In this work, we consider the egocentric video setting and propose three new text-audio retrieval benchmarks based on the EpicMIR and EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining audio-centric descriptions gives significantly higher zero-shot performance than using the original visual-centric descriptions. Furthermore, we show that using the same prompts, we can successfully employ LLMs to improve the retrieval on EpicSounds, compared to using the original audio class labels of the dataset. Finally, we confirm that LLMs can be used to determine the difficulty of identifying the action associated with a sound.
Paper Structure (17 sections, 2 figures, 10 tables)

This paper contains 17 sections, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Datasets and approach
  4. Datasets and tasks
  5. Audio description generation methodology
  6. Our proposed text-audio retrieval benchmarks
  7. Determining the audio relevancy using LLMs
  8. Experiments
  9. Conclusion
  10. Supplementary Material: A SOUND APPROACH: USING LARGE LANGUAGE MODELS TO GENERATE AUDIO DESCRIPTIONS FOR EGOCENTRIC TEXT-AUDIO RETRIEVAL
  11. EpicSounds
  12. Prompts
  13. Multimodal text-video retrieval
  14. Leveraging models finetuned on Clotho for tasks with original visual descriptions
  15. Intra-video vs inter-video results
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Frames from the EpicKitchens dataset together with the corresponding original visual descriptions from EpicMIR shown above and those generated with our approach (using the ChatGPT LLM chatgpt) below.
  • Figure 2: Given visual-centric descriptions, we propose to use an LLM (ChatGPT) to generate audio descriptions (step 3). The LLM is prompted with a task description (step 1) and few-shot paired examples of visual-centric and audio descriptions (step 2).