Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration

TL;DR

Omni-R1 introduces a two-system reinforcement learning framework to tackle the trade-off between long-horizon temporal reasoning and fine-grained pixel grounding in omnimodal video inputs. System 1 (Global Reasoning) selects informative keyframes and reformulates tasks for System 2 (Detail Understanding), which performs high-resolution grounding on the chosen snippets; the systems collaborate through end-to-end GRPO with hierarchical rewards. The approach, evaluated on RefAVS and REVOS, outperforms strong supervised baselines and specialized SOTA models, while improving out-of-domain generalization and reducing multimodal hallucination. Ablation studies and diagnostics validate the reward design and the benefits of temporal reasoning, suggesting a scalable path toward universally capable omnimodal foundation models.

Abstract

Long-horizon video-audio reasoning and fine-grained pixel understanding impose conflicting requirements on omnimodal models: dense temporal coverage demands many low-resolution frames, whereas precise grounding calls for high-resolution inputs. We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informative keyframes and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding on the selected high-resolution snippets. Because ``optimal'' keyframe selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy Optimization. Omni-R1 trains the Global Reasoning System through hierarchical rewards obtained via online collaboration with the Detail Understanding System, requiring only one epoch of RL on small task splits. Experiments on two challenging benchmarks, namely Referring Audio-Visual Segmentation (RefAVS) and Reasoning Video Object Segmentation (REVOS), show that Omni-R1 not only surpasses strong supervised baselines but also outperforms specialized state-of-the-art models, while substantially improving out-of-domain generalization and mitigating multimodal hallucination. Our results demonstrate the first successful application of RL to large-scale omnimodal reasoning and highlight a scalable path toward universally foundation models.

Figures (6)

• Figure 1: Overview of the proposed Omni-R1 system for collaborative video understanding. Left: Performance comparison across multiple benchmarks shows Omni-R1 significantly outperforms existing omni-modal and video-reinforced MLLMs on both segmentation-centric and reasoning-centric tasks. Top-right: Omni-R1 employs a two-stage collaborative framework, integrating a detail understanding system (for precise visual QA) and a global reasoning system (for temporal grounding and high-resolution key frame identification). Bottom: A qualitative example highlights Omni-R1’s precise spatial-temporal segmentation and reasoning in identifying object-centric actions, outperforming prior expert models (e.g., Sa2VA) in complex scenarios.
• Figure 2: Exclusively trained as System 1 on video segmentation tasks in an End-to-End RL pipeline, Omni-R1 improved general understanding capabilities.
• Figure 3: Keyframe selection and recaptioning prompt for System 1.
• Figure 4: Audio analyzing and recaptioning prompt for System 1.
• Figure 5: Visualization results of our method on three representative VOS cases. Each subfigure illustrates a different reasoning pattern.