Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FastSLM: Hierarchical Frame Q-Former for Effective Speech Modality Adaptation

Junseok Lee, Sangyong Lee, Chang-Jae Chun

TL;DR

FastSLM introduces a lightweight, efficient speech-language model that enables long-form speech understanding by compressing frame-level features with a Hierarchical Frame Querying Transformer (HFQ-Former) to roughly $1.67$ tokens per second. A three-stage training strategy—short-form ASR pretraining, long-form ASR-based adaptation, and instruction tuning—enables robust multimodal speech-language alignment with a relatively small LLM and limited FLOPs. Empirical results show competitive or state-of-the-art performance across ASR, translation, long-form summarization, and spoken QA Tasks, while substantially reducing memory and compute; the work also provides open-source bilingual Korean–English support. Overall, the approach offers a scalable, practical foundation for multimodal systems that need to reason over extended speech inputs toward more capable AI assistants.

Abstract

Recent advances in large language models (LLMs) have demonstrated human-expert-level capabilities, driving significant interest in their potential for achieving artificial general intelligence (AGI). In particular, there is growing momentum in adapting LLMs to various modalities, including vision, video, and speech, through the development of multimodal LLMs (MLLMs). However, existing speech-language model (SLM) research has largely overlooked cost-effective adaptation strategies for leveraging LLMs in the speech domain. In this paper, we propose FastSLM, a lightweight yet efficient SLM designed for effective understanding and reasoning over long-form speech. To address the challenge of aligning high-frame-rate speech features with LLMs, we introduce the Hierarchical Frame Querying Transformer (HFQ-Former), which compresses frame-level speech features while capturing both local and global context. Furthermore, we present a novel three-stage training strategy that enhances generalization across a wide range of speech-related tasks. Experimental results demonstrate that FastSLM achieves competitive performance compared to existing state-of-the-art models, despite operating with significantly lower FLOPs and parameter counts, while representing speech with only 1.67 tokens per second. The source code and model checkpoints are available at https://huggingface.co/okestro-ai-lab/FastSLM.

FastSLM: Hierarchical Frame Q-Former for Effective Speech Modality Adaptation

TL;DR

FastSLM introduces a lightweight, efficient speech-language model that enables long-form speech understanding by compressing frame-level features with a Hierarchical Frame Querying Transformer (HFQ-Former) to roughly tokens per second. A three-stage training strategy—short-form ASR pretraining, long-form ASR-based adaptation, and instruction tuning—enables robust multimodal speech-language alignment with a relatively small LLM and limited FLOPs. Empirical results show competitive or state-of-the-art performance across ASR, translation, long-form summarization, and spoken QA Tasks, while substantially reducing memory and compute; the work also provides open-source bilingual Korean–English support. Overall, the approach offers a scalable, practical foundation for multimodal systems that need to reason over extended speech inputs toward more capable AI assistants.

Abstract

Recent advances in large language models (LLMs) have demonstrated human-expert-level capabilities, driving significant interest in their potential for achieving artificial general intelligence (AGI). In particular, there is growing momentum in adapting LLMs to various modalities, including vision, video, and speech, through the development of multimodal LLMs (MLLMs). However, existing speech-language model (SLM) research has largely overlooked cost-effective adaptation strategies for leveraging LLMs in the speech domain. In this paper, we propose FastSLM, a lightweight yet efficient SLM designed for effective understanding and reasoning over long-form speech. To address the challenge of aligning high-frame-rate speech features with LLMs, we introduce the Hierarchical Frame Querying Transformer (HFQ-Former), which compresses frame-level speech features while capturing both local and global context. Furthermore, we present a novel three-stage training strategy that enhances generalization across a wide range of speech-related tasks. Experimental results demonstrate that FastSLM achieves competitive performance compared to existing state-of-the-art models, despite operating with significantly lower FLOPs and parameter counts, while representing speech with only 1.67 tokens per second. The source code and model checkpoints are available at https://huggingface.co/okestro-ai-lab/FastSLM.
Paper Structure (25 sections, 7 equations, 5 figures, 12 tables)

This paper contains 25 sections, 7 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio and Speech Language Models
  4. Q-Former for Speech Modality Adaptation
  5. Methodology
  6. Model Architecture of FastSLM
  7. Three-Stage Training Strategy
  8. Experiment Results
  9. Dataset Description
  10. Experimental Setup
  11. Comparison with Baseline Q-Former
  12. Quantitative Results
  13. Ablation Study
  14. Conclusion
  15. Qualitative Analysis of HFQ-Former Attention Map
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Architecture of FastSLM.
  • Figure 2: The proposed flowchart of HFQ-Former. where, $\text{C}$ denotes the concatenation.
  • Figure 3: ASR performance of FastSLM with various speech tokens. (left) LS-clean decoding result, and (right) Voxpopuli decoding result.
  • Figure 4: Comparison VRAM usage and time-to-first-token (TTFT) according to speech length.
  • Figure 5: Visualization of HFQ-Former attention patterns across speech duration (logarithmic scale). Top: short-form speech ($<30\,\mathrm{s}$). Middle: mid-form speech ($<60\,\mathrm{s}$). Bottom: long-form speech ($>15\,\mathrm{min}$).