ContextNav: Towards Agentic Multimodal In-Context Learning

TL;DR

ContextNav tackles robustness and scalability in multimodal in-context learning by introducing an agentic, closed-loop workflow that unifies context management, noise-robust contextualization, and graph-driven orchestration. It combines resource-aware multimodal embedding, adaptive retrieval, coherence and structure-aware filtering, and memory-guided workflow optimization via an Operational Grammar Graph. Empirical results show ContextNav achieving substantial ICL gains across diverse datasets and models, outperforming manual and retrieval-only baselines and highlighting the value of agentic reasoning for robust contextualization. The work underscores the potential of agentic workflows to enable scalable, adaptive, and noise-resilient multimodal ICL in real-world vision-language tasks.

Abstract

Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt to novel vision-language tasks from a few contextual examples. However, existing ICL approaches face challenges in reconciling scalability with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity-based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and noise-robust contextualization within a closed-loop workflow driven by graph-based orchestration. Specifically, it builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. An Operational Grammar Graph (OGG) further supports adaptive workflow planning and optimization, enabling the agent to refine its operational strategies based on downstream ICL feedback. Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualization in multimodal ICL.

Figures (9)

• Figure 1: Motivation for introducing agentic contextualization in multimodal ICL. Similarity-based retrieval can introduce semantic or structural noise into contextual candidates, which degrades ICL effectiveness. Employing an agent for human-like curation could effectively alleviate this challenge. We provide a further discussion of the negative effects of such noise in Section \ref{['sec:dis']}, grounded in quantitative results.
• Figure 2: Framework of the ContextNav. The proposed agentic framework integrates three synergistic modules: (a) Agentic Context Management, which performs resource-aware multimodal embedding, maintains a continuously updated vector database, and subsequently leverages it for similarity-based retrieval to generate the initial candidate pool given an input query; (b) Noise-Robust Contextualization, which refines retrieved candidates through agentic retrieval and structural alignment to mitigate both semantic and structural noise; and (c) Graph-driven Workflow Orchestration, where the agent leverages an Operational Grammar Graph and memory module to adaptively plan and optimize operation sequence, thereby controlling the workflow. Collectively, these modules enable systematic management, representation, retrieval, and organization of multimodal contexts, supporting noise-robust and dynamically optimized contextualization for multimodal ICL.
• Figure 3: Comparison of average ICL gains with baselines. (a) Dataset-wise average gains across 8 datasets. (b) Model-wise average gains across 6 representative MLLMs. 'ICL Gains' represents the percentage improvement of a vanilla MLLM after applying ICL. The complete comparative results are provided in Appendix \ref{['appendix_gains']}.
• Figure 4: Ablation study on the number of contextual shots. 'ICL Gains’ represents the percentage improvement of a vanilla MLLM after ICL.
• Figure 5: Effect of Structural Alignment on textual similarity distributions. (a) Paired scatter between original and aligned similarities. (b) Per-query distributions of similarity gains, with orange denoting positive shifts and gray denoting non-positive shifts, and the points indicate the average gains. (c) Cumulative Distribution Function (CDF) of similarity gains, with reference lines indicating zero‐gain boundaries.