MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning

TL;DR

MedAlign tackles the core obstacles in deploying large vision-language models for medical visual question answering by introducing a multimodal Direct Preference Optimization (mDPO) objective, a Retrieval-Aware Mixture-of-Experts (RA-MoE) routing mechanism, and a federated governance framework for adaptive chain-of-thought reasoning. The mDPO objective directly enforces visual grounding through cross-modal preferences and an anchor-based stabilization term, while RA-MoE activates domain-specific experts guided by multimodal retrieval signals. The federated governance module uses a meta-cognitive uncertainty estimator and a quorum-based halting and synthesis process to achieve efficient, robust reasoning across institutions. Empirical results on MIMIC-CXR, IU-Xray, and Harvard-FairVLMed show state-of-the-art performance and substantial reductions in reasoning length, with strong robustness to noise, highlighting MedAlign’s practical potential for scalable, cross-institution clinical AI services.

Abstract

Recently, large models have shown significant potential for smart healthcare. However, the deployment of Large Vision-Language Models (LVLMs) for clinical services is currently hindered by three critical challenges: a tendency to hallucinate answers not grounded in visual evidence, the inefficiency of fixed-depth reasoning, and the difficulty of multi-institutional collaboration. To address these challenges, in this paper, we develop MedAlign, a novel framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA). Specifically, we first propose a multimodal Direct Preference Optimization (mDPO) objective to explicitly align preference learning with visual context. We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM (i.e., an expert), thereby mitigating hallucinations in LVLMs. To achieve adaptive reasoning and facilitate multi-institutional collaboration, we propose a federated governance mechanism, where the selected expert, fine-tuned on clinical datasets based on mDPO, locally performs iterative Chain-of-Thought (CoT) reasoning via the local meta-cognitive uncertainty estimator. Extensive experiments on three representative Med-VQA datasets demonstrate that MedAlign achieves state-of-the-art performance, outperforming strong retrieval-augmented baselines by up to $11.85\%$ in F1-score, and simultaneously reducing the average reasoning length by $51.60\%$ compared with fixed-depth CoT approaches.

Figures (11)

• Figure 1: The overview of the LVLM inference paradigm in MedAlign, which operates in two phases. Phase 1: Retrieval-aware expert routing. The input query $(I, Q)$ is encoded into a multimodal embedding, which is used to perform parallel retrieval across $D$ domain knowledge bases. The resulting relevance scores serve as a zero-shot gating signal to select the optimal expert $E_{d*}$, while the retrieved textual context from the same operation provides additional knowledge augmentation. Phase 2: federated governance for adaptive reasoning. The selected expert then performs iterative CoT reasoning in parallel across $N$ federated sites. At each step, a local meta-cognitive uncertainty estimator $g_{\psi}$ determines whether to continue reasoning or to report its confident result to a central server. The server aggregates these high-confidence reasoning chains and synthesizes the final answer $A^*$ once a quorum consensus is achieved.
• Figure 2:
• Figure 3:
• Figure 6:
• Figure 7: