Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphs

TL;DR

The paper tackles open-domain multimodal document retrieval by modeling graph traversal over a layered component graph as an information-state MDP. It introduces Failure is Feedback FiF, an agentic retriever with a history-aware backtracking mechanism and a cost-aware orchestrator that escalates reasoning only when needed. FiF maintains a structured traversal memory and uses failure traces to re-anchor searches and revise subqueries, achieving state-of-the-art results on three benchmarks for both retrieval and end-to-end QA. Empirical results show FiF delivering higher recall and QA accuracy at the cost of increased latency and API usage, highlighting a practical tradeoff between robustness and computation. The work advances robust multimodal retrieval by integrating structured graph navigation, adaptive reasoning, and failure-driven feedback into a unified framework.

Abstract

Open-domain multimodal document retrieval aims to retrieve specific components (paragraphs, tables, or images) from large and interconnected document corpora. Existing graph-based retrieval approaches typically rely on a uniform similarity metric that overlooks hop-specific semantics, and their rigid pre-defined plans hinder dynamic error correction. These limitations suggest that a retriever should adapt its reasoning to the evolving context and recover intelligently from dead ends. To address these needs, we propose Failure is Feedback (FiF), which casts subgraph retrieval as a sequential decision process and introduces two key innovations. (i) We introduce a history-aware backtracking mechanism; unlike standard backtracking that simply reverts the state, our approach piggybacks on the context of failed traversals, leveraging insights from previous failures. (ii) We implement an economically-rational agentic workflow. Unlike conventional agents with static strategies, our orchestrator employs a cost-aware traversal method to dynamically manage the trade-off between retrieval accuracy and inference costs, escalating to intensive LLM-based reasoning only when the prior failure justifies the additional computational investment. Extensive experiments show that FiF achieves state-of-the-art retrieval on the benchmarks of MultimodalQA, MMCoQA and WebQA.

Figures (3)

• Figure 1: Motivating examples of multihop retrieval failures in existing graph retrieval approaches. (a) Vector-similarity-driven traversal follows a spurious cue. (b) Fixed retrieval plan produces an underspecified hop and fails to recover from a dead end.
• Figure 2: Visualization of an example corpus $\mathcal{D}$ and its corresponding layered component graph $\mathcal{G}$.
• Figure 3: Overview of Failure is Feedback. (a) High-level orchestration loop and action interactions, labeled using the notation from the problem formulation (§ \ref{['sec:probform']}). (b) An example layered component graph $\mathcal{G}$. Note that only the document and component nodes are shown for brevity. (c) An example memory $M_t$ of a traversal over $\mathcal{G}$.