MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading

TL;DR

MTGA introduces a dual-view representation for event-based lip-reading by combining time-segmented event frames with a voxel graph list to preserve intra-frame temporal details. A temporal granularity aligned fusion module coherently fuses global frame features with local graph features within each temporal segment, while a positional-encoding–enabled temporal backend (Bi-GRU followed by Self-Attention) captures both local and global temporal dynamics. Experiments on DVS-Lip and DVS128-Gait-Day demonstrate that MTGA outperforms state-of-the-art event-based and video-based methods, validating the effectiveness of the two-view approach and the fusion strategy. The results indicate MTGA’s potential to enhance fine-grained lip-reading and generalize to other event recognition tasks, with implications for robust silent-speech interfaces.

Abstract

Lip-reading is to utilize the visual information of the speaker's lip movements to recognize words and sentences. Existing event-based lip-reading solutions integrate different frame rate branches to learn spatio-temporal features of varying granularities. However, aggregating events into event frames inevitably leads to the loss of fine-grained temporal information within frames. To remedy this drawback, we propose a novel framework termed Multi-view Temporal Granularity aligned Aggregation (MTGA). Specifically, we first present a novel event representation method, namely time-segmented voxel graph list, where the most significant local voxels are temporally connected into a graph list. Then we design a spatio-temporal fusion module based on temporal granularity alignment, where the global spatial features extracted from event frames, together with the local relative spatial and temporal features contained in voxel graph list are effectively aligned and integrated. Finally, we design a temporal aggregation module that incorporates positional encoding, which enables the capture of local absolute spatial and global temporal information. Experiments demonstrate that our method outperforms both the event-based and video-based lip-reading counterparts.

Figures (4)

• Figure 1: The two event representations we adopt. (Left) An event frame is obtained by integrating event points within a certain temporal range in the event stream. (Right) After dividing the event stream into a voxel grid in three-dimensional space, a three-dimensional geomeric neighboring graph is constructed at regular time intervals, resulting in a voxel graph list used for event representation.
• Figure 2: The architecture of our proposed network. The model is divided into three components: (1) Event representation, which describes the events from two different viewpoints, i.e., event frames and voxel graph list; (2) Feature extraction and fusion, which extracts features for the two views separately, and combines them through a temporal granularity aligned fusion module; (3) Temporal backend network, which aggregates the global temporal information.
• Figure 3: Illustrations of our designed two modules. (Left) The temporal granularity aligned fusion module. At each time step, the voxel graph node features are convolved to the same shape and then merged. The merged features achieve the fusion of spatio-temporal features through convolution and residual block. (Right) The temporal aggregation module. Extract features from the original voxel according to encoding, and then concatenate them with the sequence model, aggregating temporal information through Bi-GRU and Self-Attention layers.
• Figure 4: Visualization of the saliency maps for words (a) “education” and (b) “tomorrow”.