Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego

TL;DR

This work demonstrates that a text-only multilingual LLM can be transformed into a cross-modal and cross-lingual retrieval system by discretizing speech into audio tokens and aligning them with text in a shared embedding space using a dual-encoder setup. The model, PaLM 2 DE, extends the LLM with an audio token vocabulary and trains with a bi-directional contrastive loss, achieving strong speech-to-text retrieval across 102 languages while training on only 21 languages. It also shows cross-lingual speech-to-text translation capabilities, which are further boosted by incorporating readily available machine translation data (WikiMatrix). The results indicate substantial performance gains over prior speech-text retrieval systems, including zero-shot cross-lingual transfer, enabling practical cross-language retrieval without extensive speech data or dual-pretraining.

Abstract

Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

TL;DR

This work demonstrates that a text-only multilingual LLM can be transformed into a cross-modal and cross-lingual retrieval system by discretizing speech into audio tokens and aligning them with text in a shared embedding space using a dual-encoder setup. The model, PaLM 2 DE, extends the LLM with an audio token vocabulary and trains with a bi-directional contrastive loss, achieving strong speech-to-text retrieval across 102 languages while training on only 21 languages. It also shows cross-lingual speech-to-text translation capabilities, which are further boosted by incorporating readily available machine translation data (WikiMatrix). The results indicate substantial performance gains over prior speech-text retrieval systems, including zero-shot cross-lingual transfer, enabling practical cross-language retrieval without extensive speech data or dual-pretraining.

Abstract

Large language models (LLMs) are trained on text-only data that go far beyond the languages with paired speech and text data. At the same time, Dual Encoder (DE) based retrieval systems project queries and documents into the same embedding space and have demonstrated their success in retrieval and bi-text mining. To match speech and text in many languages, we propose using LLMs to initialize multi-modal DE retrieval systems. Unlike traditional methods, our system doesn't require speech data during LLM pre-training and can exploit LLM's multilingual text understanding capabilities to match speech and text in languages unseen during retrieval training. Our multi-modal LLM-based retrieval system is capable of matching speech and text in 102 languages despite only training on 21 languages. Our system outperforms previous systems trained explicitly on all 102 languages. We achieve a 10% absolute improvement in Recall@1 averaged across these languages. Additionally, our model demonstrates cross-lingual speech and text matching, which is further enhanced by readily available machine translation data.
Paper Structure (23 sections, 1 equation, 3 figures, 6 tables)

This paper contains 23 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Generating Audio Tokens
  4. Supporting Text and Audio Tokens
  5. Data and Tasks
  6. Speech-to-Text Retrieval (S2T)
  7. Speech-to-Text Translation Retrieval (S2TT)
  8. Model
  9. Experiments
  10. Speech-to-Text Retrieval
  11. Seen-Unseen Breakdown
  12. Language Group Breakdown
  13. Evaluating on Cross-Modal and Cross-Lingual Tasks
  14. Zero-Shot S2TT
  15. Improving S2TT with MT Data
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Our dual encoder architecture and training pipeline. We expand the embedding layer of our backbone LLM to support the additional discretized speech tokens, that are extracted from a pre-trained speech encoder. At the same time, we tokenize the corresponding transcripts with the LLM tokenizer. We encode the speech tokens and transcripts separately and train the model with a contrastive loss over the dot product between speech and transcript embeddings.
  • Figure 2: R@1 transcription retrieval for seen and unseen languages in the training set.
  • Figure 3: BLEU scores for FLEURS zero-shot S2TT when training on Transcripts or Transcripts + Translations for PaLM 2 DE. Combining transcripts and translation data improves zero-shot S2TT retrieval.