Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies

José-M. Acosta-Triana, David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

TL;DR

The paper addresses the lack of annotated audiovisual resources for thousands of languages by introducing AnnoTheia, a semi-automatic, open-source toolkit for detecting active speakers and generating transcriptions to support language-specific data creation. It demonstrates language adaptation by fine-tuning the TalkNet-ASD model on Spanish using the ASD-RTVE corpus derived from LIP-RTVE, detailing dataset construction, training, and evaluation. The approach combines scene analysis, robust face processing, ASD using audio-visual fusion, and Whisper-based transcription to streamline annotation workflows. Overall, AnnoTheia offers a scalable path to expand audio-visual speech technologies to low-resource languages, with practical impact for data collection, annotation efficiency, and language coverage.

Abstract

More than 7,000 known languages are spoken around the world. However, due to the lack of annotated resources, only a small fraction of them are currently covered by speech technologies. Albeit self-supervised speech representations, recent massive speech corpora collections, as well as the organization of challenges, have alleviated this inequality, most studies are mainly benchmarked on English. This situation is aggravated when tasks involving both acoustic and visual speech modalities are addressed. In order to promote research on low-resource languages for audio-visual speech technologies, we present AnnoTheia, a semi-automatic annotation toolkit that detects when a person speaks on the scene and the corresponding transcription. In addition, to show the complete process of preparing AnnoTheia for a language of interest, we also describe the adaptation of a pre-trained model for active speaker detection to Spanish, using a database not initially conceived for this type of task. The AnnoTheia toolkit, tutorials, and pre-trained models are available on GitHub.

AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies

TL;DR

The paper addresses the lack of annotated audiovisual resources for thousands of languages by introducing AnnoTheia, a semi-automatic, open-source toolkit for detecting active speakers and generating transcriptions to support language-specific data creation. It demonstrates language adaptation by fine-tuning the TalkNet-ASD model on Spanish using the ASD-RTVE corpus derived from LIP-RTVE, detailing dataset construction, training, and evaluation. The approach combines scene analysis, robust face processing, ASD using audio-visual fusion, and Whisper-based transcription to streamline annotation workflows. Overall, AnnoTheia offers a scalable path to expand audio-visual speech technologies to low-resource languages, with practical impact for data collection, annotation efficiency, and language coverage.

Abstract

More than 7,000 known languages are spoken around the world. However, due to the lack of annotated resources, only a small fraction of them are currently covered by speech technologies. Albeit self-supervised speech representations, recent massive speech corpora collections, as well as the organization of challenges, have alleviated this inequality, most studies are mainly benchmarked on English. This situation is aggravated when tasks involving both acoustic and visual speech modalities are addressed. In order to promote research on low-resource languages for audio-visual speech technologies, we present AnnoTheia, a semi-automatic annotation toolkit that detects when a person speaks on the scene and the corresponding transcription. In addition, to show the complete process of preparing AnnoTheia for a language of interest, we also describe the adaptation of a pre-trained model for active speaker detection to Spanish, using a database not initially conceived for this type of task. The AnnoTheia toolkit, tutorials, and pre-trained models are available on GitHub.
Paper Structure (12 sections, 2 figures, 3 tables)

This paper contains 12 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The AnnoTheia Toolkit
  3. Fine-Tuning an ASD Model
  4. The TalkNet-ASD Model
  5. The ASD-RTVE corpus
  6. Implementation Details
  7. Results & Discussion
  8. Conclusions & Future Work
  9. Limitations
  10. Ethical Considerations
  11. Acknowledgements
  12. Bibliographical References

Figures (2)

  • Figure 1: AnnoTheia toolkit user interface. A: Video display of the scene candidate to be a new sample of the future database. An overlying green bounding box highlights the active speaker detected by the toolkit. B: Keyword legend to control the video display. C: Transcription automatically generated by the toolkit. It can be edited by the annotator. D: Buttons to allow the annotator to accept or discard the candidate scene sample. E: Navigation buttons through candidate scenes. It can be useful to correct possible annotation mistakes.
  • Figure 2: The overall architecture of the TalkNet-ASD model.