Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding

Xueyun Tian, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen

TL;DR

ROMA tackles the challenge of real-time omni-mmodal streaming by unifying proactive monitoring and reactive understanding. It introduces one-second aligned multimodal units, a time-aware RoPE scheme (TMRoPE), and a Speak Head that decouples decision timing from content generation. A two-stage fine-tuning curriculum and a curated streaming dataset enable robust cross-modal streaming adaptation and proactive responsiveness. Unified evaluation across proactive and reactive settings demonstrates state-of-the-art performance on proactive tasks and competitive results in reactive QA, highlighting ROMA’s potential for real-time, end-to-end streaming AV understanding.

Abstract

Recent Omni-multimodal Large Language Models show promise in unified audio, vision, and text modeling. However, streaming audio-video understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA, a real-time omni-multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized multimodal units, aligning dense audio with discrete video frames to handle granularity mismatches. For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12 benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-multimodal understanding.

ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding

TL;DR

ROMA tackles the challenge of real-time omni-mmodal streaming by unifying proactive monitoring and reactive understanding. It introduces one-second aligned multimodal units, a time-aware RoPE scheme (TMRoPE), and a Speak Head that decouples decision timing from content generation. A two-stage fine-tuning curriculum and a curated streaming dataset enable robust cross-modal streaming adaptation and proactive responsiveness. Unified evaluation across proactive and reactive settings demonstrates state-of-the-art performance on proactive tasks and competitive results in reactive QA, highlighting ROMA’s potential for real-time, end-to-end streaming AV understanding.

Abstract

Recent Omni-multimodal Large Language Models show promise in unified audio, vision, and text modeling. However, streaming audio-video understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA, a real-time omni-multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized multimodal units, aligning dense audio with discrete video frames to handle granularity mismatches. For online decision-making, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12 benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-multimodal understanding.
Paper Structure (55 sections, 3 equations, 12 figures, 10 tables)

This paper contains 55 sections, 3 equations, 12 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Reactive Models
  4. Proactive Models
  5. Streaming Video Understanding Benchmarks
  6. Method
  7. Model Architecture
  8. Multimodal units for temporally aligned streaming inputs.
  9. Chunk-Level Temporal Position Encoding
  10. Speak Head
  11. Training and Inference Pipeline
  12. Dataset Construction
  13. Online Proactive (27K)
  14. Online Narration (109K)
  15. Reactive QA (540K)
  16. ...and 40 more sections

Figures (12)

  • Figure 1: ROMA's streaming understanding capabilities. It supports proactive tasks, including event alerts and narration, alongside reactive question answering.
  • Figure 2: Model Architecture. Streaming inputs are processed as aligned multimodal units. The speak head determines response timing, activating the LM head (illustrated via narration) upon crossing a probability threshold.
  • Figure 3: Chunked TMRoPE. Seamlessly extends the global timeline to streaming inputs by assigning cumulative positional IDs across discrete units.
  • Figure 4: Overview of ROMA's Streaming Dataset. Statistics, task taxonomy, and sample formats.
  • Figure 5: Sensitivity analysis on window size on QVHighlight.
  • ...and 7 more figures