Make a Video Call with LLM: A Measurement Campaign over Five Mainstream Apps
Jiayang Xu, Xiangjie Huang, Zijie Li, Zili Meng
TL;DR
This work presents the first systematic benchmark for AI video chat, evaluating five mainstream apps across four dimensions—quality, latency, internal mechanisms, and system overhead—using cloud and local testbeds over multiple regions and days. It reveals a persistent trade-off between response quality and latency, highlights differences in input modalities (end-to-end vs cascaded) and network protocols (RTP/RTCP vs QUIC), and identifies bottlenecks such as memory behavior, scheduling, and bandwidth sensitivity. The study provides a real-world baseline, curated datasets (including streaming-focused and AI-RTC task datasets), and repeatable testbeds to support future optimization of AI video chat systems and to inform designers of system-level trade-offs. Overall, the results emphasize practical challenges in achieving seamless, proactive AI video conversations and offer concrete directions for improving memory management, protocol choices, and resource efficiency in RTC-based AI assistants.
Abstract
In 2025, Large Language Model (LLM) services have launched a new feature -- AI video chat -- allowing users to interact with AI agents via real-time video communication (RTC), just like chatting with real people. Despite its significance, no systematic study has characterized the performance of existing AI video chat systems. To address this gap, this paper proposes a comprehensive benchmark with carefully designed metrics across four dimensions: quality, latency, internal mechanisms, and system overhead. Using custom testbeds, we further evaluate five mainstream AI video chatbots with this benchmark. This work provides the research community a baseline of real-world performance and identifies unique system bottlenecks. In the meantime, our benchmarking results also open up several research questions for future optimizations of AI video chatbots.