Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spoken question answering for visual queries

Nimrod Shabtay, Zvi Kons, Avihu Dekel, Hagai Aronowitz, Ron Hoory, Assaf Arbelle

TL;DR

This work tackles Spoken Visual Question Answering (SVQA) by fusing text, image, and speech inputs in a LLaVA-based multimodal model with modality-specific encoders and projectors aligned to the LLM. It addresses the lack of SVQA datasets by synthesizing large-scale speech data from textual VQA resources using zero-shot TTS systems (StyleTTS2 and F5-TTS) and pretraining a speech projector before LoRA-based fine-tuning; the approach achieves strong performance and demonstrates robustness to different TTS systems. Compared to an ASR-based VQA pipeline, the SVQA models maintain competitive accuracy across diverse benchmarks, illustrating the practicality of end-to-end spoken reasoning over images. This work contributes a scalable SVQA architecture, a data-generation methodology, and practical insights into cross-modal learning with synthetic speech, representing an exceptional step toward natural human-machine interaction.

Abstract

Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy.

Spoken question answering for visual queries

TL;DR

This work tackles Spoken Visual Question Answering (SVQA) by fusing text, image, and speech inputs in a LLaVA-based multimodal model with modality-specific encoders and projectors aligned to the LLM. It addresses the lack of SVQA datasets by synthesizing large-scale speech data from textual VQA resources using zero-shot TTS systems (StyleTTS2 and F5-TTS) and pretraining a speech projector before LoRA-based fine-tuning; the approach achieves strong performance and demonstrates robustness to different TTS systems. Compared to an ASR-based VQA pipeline, the SVQA models maintain competitive accuracy across diverse benchmarks, illustrating the practicality of end-to-end spoken reasoning over images. This work contributes a scalable SVQA architecture, a data-generation methodology, and practical insights into cross-modal learning with synthetic speech, representing an exceptional step toward natural human-machine interaction.

Abstract

Question answering (QA) systems are designed to answer natural language questions. Visual QA (VQA) and Spoken QA (SQA) systems extend the textual QA system to accept visual and spoken input respectively. This work aims to create a system that enables user interaction through both speech and images. That is achieved through the fusion of text, speech, and image modalities to tackle the task of spoken VQA (SVQA). The resulting multi-modal model has textual, visual, and spoken inputs and can answer spoken questions on images. Training and evaluating SVQA models requires a dataset for all three modalities, but no such dataset currently exists. We address this problem by synthesizing VQA datasets using two zero-shot TTS models. Our initial findings indicate that a model trained only with synthesized speech nearly reaches the performance of the upper-bounding model trained on textual QAs. In addition, we show that the choice of the TTS model has a minor impact on accuracy.

Paper Structure

This paper contains 15 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Spoken VQA model architecture
  4. Datasets
  5. Speech-only datasets
  6. Speech and images datasets
  7. Model training
  8. Speech projector pre-training
  9. Supervised fine-tuning
  10. Results
  11. Spoken VQA
  12. The effect of a spoken post-prompt
  13. Word error rates
  14. Discussion
  15. Guidelines for document review and summary

Figures (1)

  • Figure 1: SVQA model components. Frozen modules are marked with a snow icon and trainable modules with a fire icon.