Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering

Yifan Zhu, Xinyu Mu, Tao Feng, Zhonghong Ou, Yuning Gong, Haoran Luo

TL;DR

OmniRAG-Agent tackles low-resource long-horizon omnimodal QA by combining retrieval-augmented generation with an agentic multi-turn reasoning loop and end-to-end GRPO optimization. It builds image and audio evidence banks from long video streams, enabling targeted retrieval and incremental reasoning to ground answers across modalities. Across OmniVideoBench, WorldSense, and Daily-Omni, OmniRAG-Agent consistently outperforms baselines and its ablations confirm the contribution of RAG, agentic planning, and RL. The approach offers a plug-and-play, budget-aware solution for leveraging OmniLLMs in real-world multimodal QA scenarios.

Abstract

Long-horizon omnimodal question answering answers questions by reasoning over text, images, audio, and video. Despite recent progress on OmniLLMs, low-resource long audio-video QA still suffers from costly dense encoding, weak fine-grained retrieval, limited proactive planning, and no clear end-to-end optimization.To address these issues, we propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning. It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks. Moreover, it uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries. Furthermore, we apply group relative policy optimization to jointly improve tool use and answer quality over time. Experiments on OmniVideoBench, WorldSense, and Daily-Omni show that OmniRAG-Agent consistently outperforms prior methods under low-resource settings and achieves strong results, with ablations validating each component.

OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering

TL;DR

OmniRAG-Agent tackles low-resource long-horizon omnimodal QA by combining retrieval-augmented generation with an agentic multi-turn reasoning loop and end-to-end GRPO optimization. It builds image and audio evidence banks from long video streams, enabling targeted retrieval and incremental reasoning to ground answers across modalities. Across OmniVideoBench, WorldSense, and Daily-Omni, OmniRAG-Agent consistently outperforms baselines and its ablations confirm the contribution of RAG, agentic planning, and RL. The approach offers a plug-and-play, budget-aware solution for leveraging OmniLLMs in real-world multimodal QA scenarios.

Abstract

Long-horizon omnimodal question answering answers questions by reasoning over text, images, audio, and video. Despite recent progress on OmniLLMs, low-resource long audio-video QA still suffers from costly dense encoding, weak fine-grained retrieval, limited proactive planning, and no clear end-to-end optimization.To address these issues, we propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning. It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks. Moreover, it uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries. Furthermore, we apply group relative policy optimization to jointly improve tool use and answer quality over time. Experiments on OmniVideoBench, WorldSense, and Daily-Omni show that OmniRAG-Agent consistently outperforms prior methods under low-resource settings and achieves strong results, with ablations validating each component.
Paper Structure (33 sections, 6 theorems, 45 equations, 8 figures, 4 tables)

This paper contains 33 sections, 6 theorems, 45 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Method: OmniRAG-Agent
  5. Multi-Modal RAG
  6. Multi-Turn Reasoning Process
  7. RL-Based Optimization
  8. Experiments
  9. Experimental Setup
  10. Main Results (RQ1)
  11. Generalization Results (RQ2)
  12. Work with Other OmniLLMs (RQ3)
  13. Ablation study (RQ4)
  14. Budget Analysis (RQ5)
  15. Conclusion
  16. ...and 18 more sections

Key Result

Proposition 4.1

Multi-modal RAG improves the agent’s ability to solve problems.

Figures (8)

  • Figure 1: An example of OmniRAG-Agent interacting with a visual–audio modal environment. The agent answers a long-horizon question by iteratively retrieving relevant audio clips and image clips from external banks and reasoning step by step.
  • Figure 2: Comparison of different approaches for long-horizon audio-video QA under low-resource constraints: end-to-end OmniLLMs, API-based OmniAgents, LLM fine-tuning optimization, and our low-resource agent training framework with image–audio clip retrieval.
  • Figure 3: An overview of the OmniRAG-Agent framework. An OmniLLM interacts with a multi-modal retrieval environment to answer long-horizon audio-video questions through multi-turn retrieval. The image and audio retrieval banks are plug-and-play.
  • Figure 4: Generalization results across benchmarks. OVB = OmniVideoBench, WS = WorldSense, DO = Daily-Omni. “→” means train on the dataset before the arrow and test on the dataset after the arrow.
  • Figure 5: Transferability across different OmniLLM backbones on OmniVideoBench. OmniRAG-Agent is applied to five representative OmniLLMs, including two closed-source models and three open-source models.
  • ...and 3 more figures

Theorems & Definitions (12)

  • Proposition 4.1
  • proof
  • Proposition 4.2
  • proof
  • Proposition 4.3
  • proof
  • Proposition 2.1
  • proof
  • Proposition 2.2
  • proof
  • ...and 2 more