Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

Leanne Nortje, Dan Oneata, Gabriel Pirlogeanu, Herman Kamper

TL;DR

This work tackles visually prompted keyword localisation (VPKL) in real low-resource settings by removing reliance on transcriptions and introducing a transcription-free, few-shot pair mining approach. It adapts a visually grounded speech model (Loc-AttNet) that fuses a vision encoder and an audio encoder via matchmap attention, trained with a contrastive loss to align image queries with spoken utterances. The study demonstrates English VPkl performance with few-shot mined pairs, approaching transcription-based upper bounds, and extends the method to Yorùbá, revealing the challenges of language-specific representations and mining accuracy. Key findings show that few-shot mining outperforms visual-tagging baselines, but significant gains depend on language-tailored initialization (e.g., CPC) and robust mining, highlighting practical viability for real low-resource languages while outlining clear directions for improvement.

Abstract

Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.

Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings

TL;DR

This work tackles visually prompted keyword localisation (VPKL) in real low-resource settings by removing reliance on transcriptions and introducing a transcription-free, few-shot pair mining approach. It adapts a visually grounded speech model (Loc-AttNet) that fuses a vision encoder and an audio encoder via matchmap attention, trained with a contrastive loss to align image queries with spoken utterances. The study demonstrates English VPkl performance with few-shot mined pairs, approaching transcription-based upper bounds, and extends the method to Yorùbá, revealing the challenges of language-specific representations and mining accuracy. Key findings show that few-shot mining outperforms visual-tagging baselines, but significant gains depend on language-tailored initialization (e.g., CPC) and robust mining, highlighting practical viability for real low-resource languages while outlining clear directions for improvement.

Abstract

Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba.
Paper Structure (12 sections, 1 equation, 3 figures, 2 tables)

This paper contains 12 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Visually prompted keyword localisation
  3. Visually grounded speech model and loss
  4. Few-shot pair mining
  5. English experiments
  6. Experimental setup
  7. Results
  8. Low-resource experiments: Yorùbá
  9. Experimental setup
  10. Yorùbá vpkl results
  11. Discussion
  12. Conclusions

Figures (3)

  • Figure 1: The goal in visually prompted keyword localisation is to detect and locate a given query keyword (given as an image) within spoken utterances. On the right, the Yorùbá word for "boys" is "ọmọkùnrin".
  • Figure 2: Loc-AttNet consists of a vision and an audio branch connected through a localisation attention mechanism.
  • Figure 3: Qualitative samples on English (left) and Yorùbá (right). Given a query image, we show the top detected audio sample and the scores for localisation. We include the corresponding keyword for reference, but this is not seen by the model. The red dotted line denotes the similarity score threshold $\alpha$, which determines whether the query image is detected in the audio.