Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

Dongyoung Go, Taesun Whang, Chanhee Lee, Hwa-Yeon Kim, Sunghoon Park, Seunghwan Ji, Jinho Kim, Dongchan Kim, Young-Bum Kim

TL;DR

CUE-M tackles the core problem of robust multimodal information retrieval by combining Retrieval-Augmented Generation with a multi-stage pipeline that understands image context and user intent. It introduces image-enriched retrieval, intent refinement, contextual query generation, API integration, and a comprehensive safety filter, all coordinated by an automatic prompt-tuning mechanism. Empirical results on real-world and public benchmarks show CUE-M outperforms baselines in knowledge-heavy queries while maintaining safety performance on par with existing models. The work demonstrates the practical potential of a service-level, multimodal RAG framework for complex, real-world information needs where external knowledge access and safety are critical.

Abstract

The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems.

CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model

TL;DR

CUE-M tackles the core problem of robust multimodal information retrieval by combining Retrieval-Augmented Generation with a multi-stage pipeline that understands image context and user intent. It introduces image-enriched retrieval, intent refinement, contextual query generation, API integration, and a comprehensive safety filter, all coordinated by an automatic prompt-tuning mechanism. Empirical results on real-world and public benchmarks show CUE-M outperforms baselines in knowledge-heavy queries while maintaining safety performance on par with existing models. The work demonstrates the practical potential of a service-level, multimodal RAG framework for complex, real-world information needs where external knowledge access and safety are critical.

Abstract

The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems.

Paper Structure

This paper contains 43 sections, 3 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal information retrieval.
  4. Retrieval-Augmented Multimodal Model.
  5. Method
  6. Image-Enriched Information Retrieval
  7. Image description.
  8. Similar image search.
  9. Image tag text search.
  10. Intention Refinement
  11. Query generation
  12. Application Selection
  13. Answer generation
  14. Safety filter
  15. Automatic prompt tuning
  16. ...and 28 more sections

Figures (3)

  • Figure 1: CUE-M is a multimodal RAG pipeline designed to interpret multimodal user intent and generate accurate answers through advanced retrieval, seamlessly integrating with diverse applications.
  • Figure 2: A high-level architecture of CUE-M pipeline, illustrating stages from image-based information retrieval and query refinement to application selection, content filtering, and final response generation. Generative models are highlighted in red, and classification models are highlighted in blue.
  • Figure 3: A high-level architecture of safety pipeline. The pipeline integrates light-weight and multimodal classifiers, with two training-free filtering approaches.