Beyond Pixels: Visual Metaphor Transfer via Schema-Driven Agentic Reasoning

TL;DR

This work defines Visual Metaphor Transfer (VMT) as autonomous extraction of cross-domain metaphor logic from a reference image and re-instantiation on a target subject. It formalizes a Schema Grammar $\mathcal{G}$ grounded in Conceptual Blending Theory and implements a closed-loop, four-agent system (Perception, Transfer, Generation, Diagnostic) to decode, adapt, and realize metaphorical meaning. Empirical results show consistent improvements over strong baselines in metaphor consistency, analogy appropriateness, and visual creativity, supported by qualitative, quantitative, and human evaluations across diverse metaphors and domains. The framework demonstrates practical impact for automated, high-quality creative content in advertising and meme generation, with public code to follow.

Abstract

A visual metaphor constitutes a high-order form of human creativity, employing cross-domain semantic fusion to transform abstract concepts into impactful visual rhetoric. Despite the remarkable progress of generative AI, existing models remain largely confined to pixel-level instruction alignment and surface-level appearance preservation, failing to capture the underlying abstract logic necessary for genuine metaphorical generation. To bridge this gap, we introduce the task of Visual Metaphor Transfer (VMT), which challenges models to autonomously decouple the "creative essence" from a reference image and re-materialize that abstract logic onto a user-specified target subject. We propose a cognitive-inspired, multi-agent framework that operationalizes Conceptual Blending Theory (CBT) through a novel Schema Grammar ("G"). This structured representation decouples relational invariants from specific visual entities, providing a rigorous foundation for cross-domain logic re-instantiation. Our pipeline executes VMT through a collaborative system of specialized agents: a perception agent that distills the reference into a schema, a transfer agent that maintains generic space invariance to discover apt carriers, a generation agent for high-fidelity synthesis and a hierarchical diagnostic agent that mimics a professional critic, performing closed-loop backtracking to identify and rectify errors across abstract logic, component selection, and prompt encoding. Extensive experiments and human evaluations demonstrate that our method significantly outperforms SOTA baselines in metaphor consistency, analogy appropriateness, and visual creativity, paving the way for automated high-impact creative applications in advertising and media. Source code will be made publicly available.

Figures (10)

• Figure 1: Diverse image metaphor transfer results generated by our framework. For each pair, the left image serves as the Reference and the right is the Generated Result. Our model demonstrates robust capability across distinct cognitive levels.
• Figure 2: Conceptual Blending Theory.
• Figure 3: Architecture of our Self-Reflective Agentic Framework for Visual Rhetoric Transfer. The system consists of Perception, Transfer, Generation, and Diagnostic agents. It transforms a reference visual metaphor ($I_{ref}$) into a new target context ($I_{gen}$) by extracting and mapping structured graph representations ($G_{ref} \to G_{tgt}$). A hierarchical feedback loop ensures the generated output faithfully preserves the rhetorical logic while adapting to the new subject matter.
• Figure 4: Qualitative comparison with baseline methods.
• Figure 5: Human evaluation study.