Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration

Ryan Soh-Eun Shim, Kwanghee Choi, Kalvin Chang, Ming-Hao Hsu, Florian Eichin, Zhizheng Wu, Alane Suhr, Michael A. Hedderich, David Harwath, David R. Mortensen, Barbara Plank

TL;DR

This work addresses script non-determinism in multilingual speech recognition by showing that script information is linearly encoded in the decoder activations of Whisper. It introduces Script Steering, a training-free method that learns per-layer script directions from activation differences and injects them at inference to control output script, enabling transliteration even in novel language-script pairings. The approach yields robust improvements across model sizes, demonstrates zero-shot generalization and one-shot efficiency, and reveals that script-based transliteration tends to be phonetic rather than purely orthographic. The findings suggest practical benefits for post-hoc script control in cross-script contexts and invite further exploration of cross-script alignment dynamics during training and across more transliteration directions.

Abstract

Multilingual speech foundation models such as Whisper are trained on web-scale data, where data for each language consists of a myriad of regional varieties. However, different regional varieties often employ different scripts to write the same language, rendering speech recognition output also subject to non-determinism in the output script. To mitigate this problem, we show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script. We find the addition of such script vectors to activations at test time can induce script change even in unconventional language-script pairings (e.g. Italian in Cyrillic and Japanese in Latin script). We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.

Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration

TL;DR

This work addresses script non-determinism in multilingual speech recognition by showing that script information is linearly encoded in the decoder activations of Whisper. It introduces Script Steering, a training-free method that learns per-layer script directions from activation differences and injects them at inference to control output script, enabling transliteration even in novel language-script pairings. The approach yields robust improvements across model sizes, demonstrates zero-shot generalization and one-shot efficiency, and reveals that script-based transliteration tends to be phonetic rather than purely orthographic. The findings suggest practical benefits for post-hoc script control in cross-script contexts and invite further exploration of cross-script alignment dynamics during training and across more transliteration directions.

Abstract

Multilingual speech foundation models such as Whisper are trained on web-scale data, where data for each language consists of a myriad of regional varieties. However, different regional varieties often employ different scripts to write the same language, rendering speech recognition output also subject to non-determinism in the output script. To mitigate this problem, we show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script. We find the addition of such script vectors to activations at test time can induce script change even in unconventional language-script pairings (e.g. Italian in Cyrillic and Japanese in Latin script). We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
Paper Structure (34 sections, 5 equations, 6 figures, 10 tables)

This paper contains 34 sections, 5 equations, 6 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Script Steering
  4. Activation collection
  5. Direction isolation
  6. Activation addition
  7. Experiments
  8. Script confusion
  9. no-prompt
  10. prompt
  11. steer (ours)
  12. Transliteration
  13. zero-shot (ours)
  14. pseudo label (ours)
  15. Evaluation
  16. ...and 19 more sections

Figures (6)

  • Figure 1: We induce script change in the transcriptions of Whisper, which is done by identifying "script directions" in activation space. Adding such directions to activations at test time induces the desired script. The example shows inducing the Italian word buongiorno to be transcribed in Cyrillic characters. Phonetic transcriptions are provided to illustrate the pronunciation of the transliterations.
  • Figure 2: Illustration of our method for extracting script vectors. (1) For each decoder layer, we collect activations in the source (yellow) and target (blue) script. (2) We isolate a script direction by subtracting the mean of the target activations from the mean of the source activations for each layer. (3) At test time, we add the script direction to the activations to induce the transcription to be in the target script.
  • Figure 3: Script confusion mitigation accuracy. Line color shows method. The x-axis shows model size, while the y-axis shows normalized edit similarity to ground truth in target script (\ref{['eq:script-similarity']}).
  • Figure 4: Romanization accuracy across different languages for Whisper large-v2. The x-axis shows languages, while the y-axis shows normalized edit similarity against ground truth in target script (\ref{['eq:script-similarity']}).
  • Figure 5: Cyrillization accuracy across different languages for Whisper large-v2. The x-axis shows languages, while the y-axis shows normalized edit similarity against ground truth in target script (\ref{['eq:script-similarity']}).
  • ...and 1 more figures