FUSE : Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation

TL;DR

This paper tackles failures in multimodal search and recommendation within agent-based pipelines by introducing FUSE, which combines a compact Graphical Design Representation (GDR), budgeted context strategies, and a pipeline attribution layer. It formalizes a five-stage subagent pipeline and develops seven context budgeting variants, with Context Compression emerging as the most effective across intent, routing, recall, and ranking, while maintaining favorable latency and cost profiles. The authors validate their approach on 788 real-world evaluation cases, showing high intent accuracy ($ ext{up to }93.3 ext{%}$), robust recall ($ ext{up to }99.4 ext{%}$), and strong ranking stability ($NDCG@5 ext{ around }0.889$), achieved at the fastest end-to-end latency ($p95 ext{ around }1.54 ext{s}$). Beyond performance, the work provides a principled Performance Attribution Framework and actionable deployment guidance, highlighting the trade-offs between context richness and operational efficiency in production-grade multimodal systems. Overall, FUSE demonstrates that targeted context summarization and systematic failure attribution can outperform both fully rich and minimal-context baselines while meeting real-time constraints in professional creative workflows.

Abstract

Multimodal creative assistants decompose user goals and route tasks to subagents for layout, styling, retrieval, and generation. Retrieval quality is pivotal, yet failures can arise at several stages: understanding user intent, choosing content types, finding candidates (recall), or ranking results. Meanwhile, sending and processing images is costly, making naive multimodal approaches impractical. We present FUSE: Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation. FUSE replaces most raw-image prompting with a compact Grounded Design Representation (GDR): a selection aware JSON of canvas elements (image, text, shape, icon, video, logo), structure, styles, salient colors, and user selection provided by the Planner team. FUSE implements seven context budgeting strategies: comprehensive baseline prompting, context compression, chain-of-thought reasoning, mini-shot optimization, retrieval-augmented context, two-stage processing, and zero-shot minimalism. Finally, a pipeline attribution layer monitors system performance by converting subagent signals into simple checks: intent alignment, content-type/routing sanity, recall health (e.g., zero-hit and top-match strength), and ranking displacement analysis. We evaluate the seven context budgeting variants across 788 evaluation queries from diverse users and design templates (refer Figure 3). Our systematic evaluation reveals that Context Compression achieves optimal performance across all pipeline stages, with 93.3% intent accuracy, 86.8% routing success(with fallbacks), 99.4% recall, and 88.5% NDCG@5. This approach demonstrates that strategic context summarization outperforms both comprehensive and minimal contextualization strategies.

Figures (6)

• Figure 1: Multi‑stage search system flow with FUSE integration. The diagram shows three operation modes: (a) raw query, (b) superagent‑enriched prompt, and (c) superagent prompt + compact GDR. The contextual subagent performs intent analysis, content‑type routing, recall, ranking, and semantic tie‑breaking. The evaluation and attribution layer extracts stage‑wise metrics, applies heuristic and LLM‑as‑judge evaluation, localizes failures via blame distribution, and drives targeted fixes and configuration updates.
• Figure 2: Dataset created based out of large scale human evaluation. The left image shows query length distribution with the average length of seven words per query(after stemming). The right image shows the most frequently occurring words(after stemming) in the query, those being add, background, replace, shape and image.
• Figure 3: Example of context-aware search in a creative design workflow. A user working on a coffee roasting poster issues the query "find beans". The raw query returns generic beans (left result panel), while the contextual search: informed by the template content and design context prioritizes coffee beans (right result panel), demonstrating how contextual grounding improves retrieval relevance.
• Figure 4: We analyze Pipeline Performance for Intent Match Rate, Routing Success Rate, Recall Success Rate and Average NDCG@5 across Context Budgeting strategies
• Figure 5: Improvement in routing success rate with fallback across Context Budgeting strategies. The strategies are ordered by most gain in performance.