LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models

TL;DR

The paper tackles the gap in multimodal LLMs for Spoken Question Answering by introducing LibriSQA, a large-scale LibriSpeech-based dataset with Part I free-form QA and Part II four-option questions with analysis. It proposes a lightweight, end-to-end framework that fuses speech and text within an LLM (LLaMA-7B with adapters) using pre-trained speech encoders and a frozen text encoder, trained with a simple NLL objective. Empirical results show strong performance, including 71.1% accuracy on Part II with minimal trainable parameters, and demonstrate benefits for both SQA and ASR tasks, along with insights on feature extractors and computation costs. The work highlights improved speech-text alignment in LLMs and offers a practical path toward universal multilingual multimodal models, with dataset and demo available for further research.

Abstract

While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features. To address the SQA challenge on LLMs, we initially curated the free-form and open-ended LibriSQA dataset from Librispeech, comprising Part I with natural conversational formats and Part II encompassing multiple-choice questions followed by answers and analytical segments. Both parts collectively include 107k SQA pairs that cover various topics. Given the evident paucity of existing speech-text LLMs, we propose a lightweight, end-to-end framework to execute the SQA task on the LibriSQA, witnessing significant results. By reforming ASR into the SQA format, we further substantiate our framework's capability in handling ASR tasks. Our empirical findings bolster the LLMs' aptitude for aligning and comprehending multimodal information, paving the way for the development of universal multimodal LLMs. The dataset and demo can be found at https://github.com/ZihanZhaoSJTU/LibriSQA.

Figures (8)

• Figure 1: Four application scenarios of our framework are described from the top to bottom. The first scenario involves a text-only Question Answering (QA) modality. The remaining scenarios are multimodal QAs, where the background information is provided through speech and the questions are presented in textual modality. The third and fourth scenarios are taken from LibriSQA Part I and Part II respectively. In the example shown here, the same speech is used, which corresponds to the following text: "A large open fireplace, with rusty dogs in it, and a bare boarded floor; at the far end, fleeces of wool stacked up; in the middle of the floor, some empty corn-bags."
• Figure 2: This figure depicts the word frequencies counted on LibriSQA.
• Figure 3: The two figures, from left to right, represent the distribution of questions for LibriSQA Part I and Part II, respectively. The larger the space occupied by a word, the higher its frequency of occurrence. Due to space limitations, some samples are not included in the statistics.
• Figure 4: The architecture of our framework. The speech shown here corresponds to the text of "A large open fireplace, with rusty dogs in it, and a bare boarded floor; at the far end, fleeces of wool stacked up; in the middle of the floor, some empty corn-bags."
• Figure 5: The left figure shows some examples of the ASR task using the model trained with the reformed SQA form of LibriSpeech. The right figure shows some examples using the model trained without using any ASR dataset, and only LibriSQA is used for training.