PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios

TL;DR

PhoStream addresses the challenge of evaluating real-time streaming for omnimodal mobile assistants by introducing a mobile-centric benchmark with long-form, open-ended QA across on-screen and off-screen content. It combines an Automated Generative Pipeline for scalable QA generation, a realistic Online Inference Pipeline for streaming interactions, and an LLM-as-a-Judge framework to assess timely, coherent responses. A key finding is a pronounced Forward task bottleneck driven by Early Response bias, where models answer before sufficient cues appear, underscoring a deficiency in temporal decision-making beyond content understanding. The benchmark, with 5,572 QA pairs from 578 videos, enables comprehensive evaluation of video, audio, and temporal reasoning and offers public data to accelerate development of reliable on-device mobile assistants.

Abstract

Multimodal Large Language Models excel at offline audio-visual understanding, but their ability to serve as mobile assistants in continuous real-world streams remains underexplored. In daily phone use, mobile assistants must track streaming audio-visual inputs and respond at the right time, yet existing benchmarks are often restricted to multiple-choice questions or use shorter videos. In this paper, we introduce PhoStream, the first mobile-centric streaming benchmark that unifies on-screen and off-screen scenarios to evaluate video, audio, and temporal reasoning. PhoStream contains 5,572 open-ended QA pairs from 578 videos across 4 scenarios and 10 capabilities. We build it with an Automated Generative Pipeline backed by rigorous human verification, and evaluate models using a realistic Online Inference Pipeline and LLM-as-a-Judge evaluation for open-ended responses. Experiments reveal a temporal asymmetry in LLM-judged scores (0-100): models perform well on Instant and Backward tasks (Gemini 3 Pro exceeds 80), but drop sharply on Forward tasks (16.40), largely due to early responses before the required visual and audio cues appear. This highlights a fundamental limitation: current MLLMs struggle to decide when to speak, not just what to say. Code and datasets used in this work will be made publicly accessible at https://github.com/Lucky-Lance/PhoStream.

Figures (10)

• Figure 1: Motivating example from PhoStream. The question is asked at 05:35, but the key evidence appears later. Gemini 3 Pro answers immediately and is wrong, while Doubao-Seed-1.6 keeps watching and answers correctly at 05:43.
• Figure 2: Distribution of data sources and QA capability in PhoStream. The dataset comprises 5,572 question-answer pairs from 578 videos spanning 4 mobile-centric scenarios (YouTube vlogs, phone tutorials, app recordings, and egocentric videos). It covers 10 distinct capabilities, such as UI navigation, action recognition, audio-visual integration, etc.
• Figure 3: Data examples in PhoStream. (Top) An on-screen scenario involving a Forward task, where the model must wait for subsequent UI operations or specific interface states. (Bottom) An off-screen scenario featuring a Backward task, requiring the model to trace back through the video history to answer questions about past events.
• Figure 4: Workflow of the PhoStream Automated Generative Pipeline. The process integrates three primary stages: video preprocessing for stabilization and segmentation, automated QA generation, and a multi-step verification process. Gemini 3 Pro is utilized as the core engine for both initial data annotation and subsequent quality checks. This approach ensures high-quality data by using precise temporal verification to refine or discard candidate QA pairs, effectively converting raw mobile video streams into verified benchmarks.
• Figure 5: Online Inference Pipeline in PhoStream. The model processes continuous video streams sequentially in 1-second intervals while maintaining a sliding memory window (e.g., 60s) to manage context. Unlike traditional approaches lin2024streamingbenchniu2025ovo that repetitively query the model at every time step, we issue the query only once at the relevant questioning timestamp, allowing the model to autonomously determine the response timing, thereby enabling a unified strategy to handle diverse temporal logics.