Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systems

TL;DR

The paper tackles real-time multilingual lip synchronization in resource-constrained IoT video systems by introducing a pipeline-parallel, asynchronous architecture that decouples translation and lip-sync using message queues. It features an ATP with semantic-aware speech segmentation and a high-throughput RVP with optimized lip-synchronization inference, coordinated by a timestamp-based orchestration layer. Key results include up to 3.1x end-to-end latency reduction, a 4.7x speedup for Wav2Lip via TensorRT FP16, and substantial gains in translation quality and perceptual MOS, demonstrated on consumer hardware. The work advances edge-friendly, low-latency multimodal communication for AIoT, enabling scalable deployment across heterogeneous devices and future extensions for multi-speaker and distributed setups.

Abstract

This paper introduces a parallel and asynchronous Transformer framework designed for efficient and accurate multilingual lip synchronization in real-time video conferencing systems. The proposed architecture integrates translation, speech processing, and lip-synchronization modules within a pipeline-parallel design that enables concurrent module execution through message-queue-based decoupling, reducing end-to-end latency by up to 3.1 times compared to sequential approaches. To enhance computational efficiency and throughput, the inference workflow of each module is optimized through low-level graph compilation, mixed-precision quantization, and hardware-accelerated kernel fusion. These optimizations provide substantial gains in efficiency while preserving model accuracy and visual quality. In addition, a context-adaptive silence-detection component segments the input speech stream at semantically coherent boundaries, improving translation consistency and temporal alignment across languages. Experimental results demonstrate that the proposed parallel architecture outperforms conventional sequential pipelines in processing speed, synchronization stability, and resource utilization. The modular, message-oriented design makes this work applicable to resource-constrained IoT communication scenarios including telemedicine, multilingual kiosks, and remote assistance systems. Overall, this work advances the development of low-latency, resource-efficient multimodal communication frameworks for next-generation AIoT systems.

Figures (2)

• Figure 1: Proposed parallel and asynchronous architecture showing the ATP, RVP, and orchestration layer with message queues for inter-module communication.
• Figure 2: End-to-end latency comparison between the baseline sequential and the proposed parallel architectures.