Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Resource-Efficient Multimodal Intelligence: Learned Routing among Specialized Expert Models

Mayank Saini, Arit Kumar Bishwas

TL;DR

The paper tackles the cost-inefficiency of deploying multimodal AI by introducing a modular, cost-aware routing framework that allocates queries to specialized expert models based on modality and predicted complexity. It integrates a learned routing network, LangGraph-driven multi-agent orchestration, and the Couplet framework to combine traditional perception modules with lightweight language-model coordination. Across benchmarks like MMLU and VQA, the approach achieves comparable or superior accuracy while reducing premium-model usage by over 67%, and it delivers improvements in latency and throughput. This work provides a scalable blueprint for resource-efficient, multimodal AI deployment and highlights directions for continual learning and expanded modalities.

Abstract

As AI moves beyond text, large language models (LLMs) increasingly power vision, audio, and document understanding; however, their high inference costs hinder real-time, scalable deployment. Conversely, smaller open-source models offer cost advantages but struggle with complex or multimodal queries. We introduce a unified, modular framework that intelligently routes each query - textual, multimodal, or complex - to the most fitting expert model, using a learned routing network that balances cost and quality. For vision tasks, we employ a two-stage open-source pipeline optimized for efficiency and reviving efficient classical vision components where they remain SOTA for sub-tasks. On benchmarks such as Massive Multitask Language Understanding (MMLU) and Visual Question Answering (VQA), we match or exceed the performance of always-premium LLM (monolithic systems with one model serving all query types) performance, yet reduce the reliance on costly models by over 67%. With its extensible, multi-agent orchestration, we deliver high-quality, resource-efficient AI at scale.

Towards Resource-Efficient Multimodal Intelligence: Learned Routing among Specialized Expert Models

TL;DR

The paper tackles the cost-inefficiency of deploying multimodal AI by introducing a modular, cost-aware routing framework that allocates queries to specialized expert models based on modality and predicted complexity. It integrates a learned routing network, LangGraph-driven multi-agent orchestration, and the Couplet framework to combine traditional perception modules with lightweight language-model coordination. Across benchmarks like MMLU and VQA, the approach achieves comparable or superior accuracy while reducing premium-model usage by over 67%, and it delivers improvements in latency and throughput. This work provides a scalable blueprint for resource-efficient, multimodal AI deployment and highlights directions for continual learning and expanded modalities.

Abstract

As AI moves beyond text, large language models (LLMs) increasingly power vision, audio, and document understanding; however, their high inference costs hinder real-time, scalable deployment. Conversely, smaller open-source models offer cost advantages but struggle with complex or multimodal queries. We introduce a unified, modular framework that intelligently routes each query - textual, multimodal, or complex - to the most fitting expert model, using a learned routing network that balances cost and quality. For vision tasks, we employ a two-stage open-source pipeline optimized for efficiency and reviving efficient classical vision components where they remain SOTA for sub-tasks. On benchmarks such as Massive Multitask Language Understanding (MMLU) and Visual Question Answering (VQA), we match or exceed the performance of always-premium LLM (monolithic systems with one model serving all query types) performance, yet reduce the reliance on costly models by over 67%. With its extensible, multi-agent orchestration, we deliver high-quality, resource-efficient AI at scale.

Paper Structure

This paper contains 18 sections, 9 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework
  4. High-Level System Architecture
  5. Detailed Workflow and Components
  6. Query Intake, Classification, and Modality Flagging
  7. Routing and Orchestration
  8. Text Query Routing
  9. Complex Multimodal and Multi-Agent System
  10. Context Agent and Memory Management
  11. Routing Feedback and Policy Adjustment
  12. Couplet Framework Integration
  13. Case Study: ChatGPT’s Video Analysis
  14. Operational Flow
  15. Evaluation and Results
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Initial modality-based routing where text queries are routed to LLMs based on complexity, while non-text inputs (e.g., image, audio, video, document) are dispatched to specialized expert pipelines.
  • Figure 2: High-Level System Architecture.
  • Figure 3: LangGraph Multi-Agent System Workflow.
  • Figure 4: Couplet Framework Execution Flow: Symmetric routing from user query to traditional models with SLM coordination.
  • Figure 5: Relative share of total processing cost for different models under two scenarios: (1) all queries routed to GPT-4 (All GPT-4), and (2) our routing approach.
  • ...and 1 more figures