PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Multimodal Agents

TL;DR

PolarMem introduces a training-free, inference-time memory intervention that grounds multimodal reasoning in verifiable evidence by converting fuzzy perceptual likelihoods into discrete logical states. It builds a polarized graph memory with explicit negative constraints and uses a lexicographical retrieval protocol to enforce logical consistency before semantic similarity. The approach unifies visual and textual knowledge via dual pathways, a polarized topology, and a logic-dominant generation process, all while remaining model-free with respect to training. Extensive evaluation across eight frozen Vision-Language Models and six benchmarks demonstrates improved verifiable grounding and reduced hallucinations, especially in retrieval-heavy, long-context scenarios.

Abstract

As multimodal agents evolve from passive observers to long-horizon decision-makers, they require memory systems that provide not just information availability but logical verifiability. A fundamental limitation of current architectures is the epistemic asymmetry inherent in probabilistic vision-language models and dense associative memories: they conflate semantic affinity with factual existence and structurally fail to encode negative constraints. To this end, we introduce PolarMem, a training-free Polarized Latent Graph Memory designed to ground agent reasoning in verifiable evidence. PolarMem transforms fuzzy perceptual likelihoods into discrete logical constraints through non-parametric distributional partitioning. Furthermore, it employs a polarized graph topology with orthogonal inhibitory connections to explicitly store verified negation as a primary cognitive state. At inference time, we enforce a logic-dominant retrieval paradigm, suppressing hallucinatory patterns that violate negative constraints. Extensive evaluation across eight frozen Vision--Language Models and six benchmarks demonstrates that PolarMem functions as a robust cognitive system, establishing a foundation for verifiable multimodal agents. Our code is available at https://github.com/czs-ict/PolarMem.

Figures (3)

• Figure 1: Performance comparison across four representative VLM families. The radar charts illustrate the performance of PolarMem (solid lines) versus the baselines (dashed lines) on six multi-modal benchmarks. We evaluate models of varying scales (indicated by circles for smaller and stars for larger models) within the (a) Qwen, (b) InternVL, (c) DeepSeek, and (d) LLaVA series. PolarMem consistently expands the performance envelope, particularly in retrieval-intensive tasks (MRAG, MRAMG).
• Figure 2: Architectural overview of the PolarMem framework.(A) Dual-Pathway Logic Construction: The Visual Pathway ($m_I$) employs ensemble semantic consistency verification and adaptive distributional partitioning to categorize candidate concepts into Positive, Negative, and Uncertain states. The Textual Pathway ($m_T$) extracts semantic entities to establish deterministic alignment edges. (B) Polarized Graph Memory ($\mathcal{G}_{polar}$): A heterogeneous topology that explicitly encodes negative knowledge via orthogonal $\mathcal{E}_{NOT\_HAS}$ edges, transforming fuzzy perceptual likelihoods into verifiable logical constraints. (C) Retrieval and Calibrated Inference: A lexicographical logic-aware retrieval protocol prioritizes logical consistency over semantic similarity, ensuring constraint dominance. The resulting logically sanitized evidence is serialized into a multimodal context, enabling the frozen VLM backbone to generate responses grounded in verifiable memory rather than hallucination-prone parametric priors.
• Figure 3: Ablation study analyzing the impact of different logical constraint configurations (Positive Only, Negative Only, Full) on retrieval accuracy and F1 score.