Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer

Young-Hu Park, Rae-Hong Park, Hyung-Min Park

TL;DR

SwinLip tackles the computational burden of visual speech encoders by leveraging a lightweight Swin Transformer and introducing a 3D Spatio-Temporal Embedding Module plus a Conformer-inspired 1D Attention block to capture temporal cues. The approach is validated across English and Mandarin lip reading benchmarks (LRW, LRW-1000, LRS2, LRS3), achieving state-of-the-art performance on Mandarin LRW-1000 with lower FLOPs and enabling streaming inference by removing certain temporal components. Key contributions include configuring a Swin Transformer for small lip-reading inputs, integrating temporal embeddings without compromising patch-based processing, and demonstrating compatibility with diverse backends while delivering substantial efficiency gains. The results support SwinLip as a practical, high-performance visual speech encoder suitable for real-time lip reading and AVSR systems, with clear potential for integration with audio-based models and knowledge-distillation frameworks.

Abstract

This paper presents an efficient visual speech encoder for lip reading. While most recent lip reading studies have been based on the ResNet architecture and have achieved significant success, they are not sufficiently suitable for efficiently capturing lip reading features due to high computational complexity in modeling spatio-temporal information. Additionally, using a complex visual model not only increases the complexity of lip reading models but also induces delays in the overall network for multi-modal studies (e.g., audio-visual speech recognition, speech enhancement, and speech separation). To overcome the limitations of Convolutional Neural Network (CNN)-based models, we apply the hierarchical structure and window self-attention of the Swin Transformer to lip reading. We configure a new lightweight scale of the Swin Transformer suitable for processing lip reading data and present the SwinLip visual speech encoder, which efficiently reduces computational load by integrating modified Convolution-augmented Transformer (Conformer) temporal embeddings with conventional spatial embeddings in the hierarchical structure. Through extensive experiments, we have validated that our SwinLip successfully improves the performance and inference speed of the lip reading network when applied to various backbones for word and sentence recognition, reducing computational load. In particular, our SwinLip demonstrated robust performance in both English LRW and Mandarin LRW-1000 datasets and achieved state-of-the-art performance on the Mandarin LRW-1000 dataset with less computation compared to the existing state-of-the-art model.

SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer

TL;DR

SwinLip tackles the computational burden of visual speech encoders by leveraging a lightweight Swin Transformer and introducing a 3D Spatio-Temporal Embedding Module plus a Conformer-inspired 1D Attention block to capture temporal cues. The approach is validated across English and Mandarin lip reading benchmarks (LRW, LRW-1000, LRS2, LRS3), achieving state-of-the-art performance on Mandarin LRW-1000 with lower FLOPs and enabling streaming inference by removing certain temporal components. Key contributions include configuring a Swin Transformer for small lip-reading inputs, integrating temporal embeddings without compromising patch-based processing, and demonstrating compatibility with diverse backends while delivering substantial efficiency gains. The results support SwinLip as a practical, high-performance visual speech encoder suitable for real-time lip reading and AVSR systems, with clear potential for integration with audio-based models and knowledge-distillation frameworks.

Abstract

This paper presents an efficient visual speech encoder for lip reading. While most recent lip reading studies have been based on the ResNet architecture and have achieved significant success, they are not sufficiently suitable for efficiently capturing lip reading features due to high computational complexity in modeling spatio-temporal information. Additionally, using a complex visual model not only increases the complexity of lip reading models but also induces delays in the overall network for multi-modal studies (e.g., audio-visual speech recognition, speech enhancement, and speech separation). To overcome the limitations of Convolutional Neural Network (CNN)-based models, we apply the hierarchical structure and window self-attention of the Swin Transformer to lip reading. We configure a new lightweight scale of the Swin Transformer suitable for processing lip reading data and present the SwinLip visual speech encoder, which efficiently reduces computational load by integrating modified Convolution-augmented Transformer (Conformer) temporal embeddings with conventional spatial embeddings in the hierarchical structure. Through extensive experiments, we have validated that our SwinLip successfully improves the performance and inference speed of the lip reading network when applied to various backbones for word and sentence recognition, reducing computational load. In particular, our SwinLip demonstrated robust performance in both English LRW and Mandarin LRW-1000 datasets and achieved state-of-the-art performance on the Mandarin LRW-1000 dataset with less computation compared to the existing state-of-the-art model.
Paper Structure (21 sections, 3 equations, 4 figures, 10 tables)

This paper contains 21 sections, 3 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Lip Reading
  4. Vision-Transformer-like Architectures
  5. Proposed Method
  6. 3D Spatio-Temporal Embedding Module
  7. Swin Transformer for Lip Reading
  8. 1D Convolutional Attention Module
  9. Experiments
  10. Datasets
  11. Data Pre-processing
  12. Training Details
  13. Results
  14. Effects of SwinLip
  15. Ablation Study
  16. ...and 6 more sections

Figures (4)

  • Figure 1: The overall structure of the proposed SwinLip model for lip reading. The model consists of a combination of a 3D Spatio-Temporal Embedded Module, Swin Transformer and a 1D Convolutional Attention Module. $P$ is the patch size. In the streaming mode, the MHSA layer is removed from the 1D Convolutional Attention Module.
  • Figure 2: Backend structures combined with the proposed SwinLip. They are temporal backend structures for lip reading of English words (LRW), Mandarin words (LRW-1000), and English sentences (LRS2/LRS3), respectively.
  • Figure 3: Inference times of ResNet18-based and SwinLip visual speech encoders according to the number of words. Measurements were made on a system including a GPU of NVIDIA RTX 3090Ti.
  • Figure 4: An example of a visual encoder structure applying other vision models to lip reading compared to the SwinLip structure. To construct a model similar to SwinLip, we introduced the 3D Spatio-Temporal Embedded Model and the Conformer block.