Development and evaluation of a deep learning algorithm for German word recognition from lip movements

TL;DR

This study introduces the first German-language lip-reading neural network by building and evaluating a 3D CNN/GRU hybrid (GRUConv) on a large, carefully curated German visual speech dataset. Through extensive preprocessing (mouth-focused cropping and multiple color spaces) and comparison of Conv3D, GRU, and GRUConv architectures, the authors demonstrate that mouth-crop inputs significantly boost accuracy, achieving up to $87.3\%$ on known speakers and $62.61\%$ on unseen speakers. The results show that dataset size and quality drive performance more than model complexity, with GRUConv offering the best balance of accuracy and generalization. The work provides a foundation for practical German visual speech recognition, with implications for accessibility and noisy-environment speech support.

Abstract

When reading lips, many people benefit from additional visual information from the lip movements of the speaker, which is, however, very error prone. Algorithms for lip reading with artificial intelligence based on artificial neural networks significantly improve word recognition but are not available for the German language. A total of 1806 video clips with only one German-speaking person each were selected, split into word segments, and assigned to word classes using speech-recognition software. In 38,391 video segments with 32 speakers, 18 polysyllabic, visually distinguishable words were used to train and validate a neural network. The 3D Convolutional Neural Network and Gated Recurrent Units models and a combination of both models (GRUConv) were compared, as were different image sections and color spaces of the videos. The accuracy was determined in 5000 training epochs. Comparison of the color spaces did not reveal any relevant different correct classification rates in the range from 69% to 72%. With a cut to the lips, a significantly higher accuracy of 70% was achieved than when cut to the entire speaker's face (34%). With the GRUConv model, the maximum accuracies were 87% with known speakers and 63% in the validation with unknown speakers. The neural network for lip reading, which was first developed for the German language, shows a very high level of accuracy, comparable to English-language algorithms. It works with unknown speakers as well and can be generalized with more word classes.

Figures (5)

• Figure 1: Example of a video which was cropped to the face (red) and lips (blue).
• Figure 2: Schema of the GRUConv model.
• Figure 3: Schema of the training strategy.
• Figure 4: Word recognition accuracy (correct classification rate) versus number of training epochs comparing (a) image crops and (b) color spaces.
• Figure 5: Confusion matrices of predicted versus true word classes for (a) Dataset A and (b) Dataset B using GRUConv. Entries show prediction frequencies for each true class (y-axis) across predicted classes (x-axis). Diagonal entries indicate correct predictions; off-diagonal entries show misclassifications.