MORE: Multi-Objective Adversarial Attacks on Speech Recognition

TL;DR

This work investigates robustness of large-scale ASR models to adversarial perturbations in both transcription accuracy and decoding efficiency. It proposes MORE, a hierarchical, multi-objective attack that first degrades accuracy (repulsion) and then elongates outputs (anchoring) through REDO and EOS suppression, formalized with dual objectives on $WER$ and transcript length. Empirical evaluations on Whisper variants show MORE yields substantially longer, highly degraded transcriptions compared to baselines, across LibriSpeech and LJ-Speech and under varying SNRs; ablations confirm the critical roles of REDO and EOS mechanisms. The findings highlight a dual vulnerability in ASR systems and motivate defenses at decoding-time (loop detectors), input-time (band-limiting), and training-time (adversarial training focusing on EOS/repetition paths), with public release of the attack framework to support robustness research.

Abstract

The emergence of large-scale automatic speech recognition (ASR) models such as Whisper has greatly expanded their adoption across diverse real-world applications. Ensuring robustness against even minor input perturbations is therefore critical for maintaining reliable performance in real-time environments. While prior work has mainly examined accuracy degradation under adversarial attacks, robustness with respect to efficiency remains largely unexplored. This narrow focus provides only a partial understanding of ASR model vulnerabilities. To address this gap, we conduct a comprehensive study of ASR robustness under multiple attack scenarios. We introduce MORE, a multi-objective repetitive doubling encouragement attack, which jointly degrades recognition accuracy and inference efficiency through a hierarchical staged repulsion-anchoring mechanism. Specifically, we reformulate multi-objective adversarial optimization into a hierarchical framework that sequentially achieves the dual objectives. To further amplify effectiveness, we propose a novel repetitive encouragement doubling objective (REDO) that induces duplicative text generation by maintaining accuracy degradation and periodically doubling the predicted sequence length. Overall, MORE compels ASR models to produce incorrect transcriptions at a substantially higher computational cost, triggered by a single adversarial input. Experiments show that MORE consistently yields significantly longer transcriptions while maintaining high word error rates compared to existing baselines, underscoring its effectiveness in multi-objective adversarial attack.

Figures (2)

• Figure 1: Overview of our proposed multi-objective repetitive encouragement doubling (MORE) adversarial attack method.
• Figure 2: Inference Time of Whisper-large versus output length.