HyperWalker: Dynamic Hypergraph-Based Deep Diagnosis for Multi-Hop Clinical Modeling across EHR and X-Ray in Medical VLMs

TL;DR

HyperWalker tackles the challenge of automated clinical diagnosis by moving beyond sample-isolated inference to structured, multi-hop reasoning that integrates longitudinal EHRs, imaging, and medical knowledge. It introduces iBrochure, a dynamic, implicit heterogeneous hypergraph, and Walker, an RL-driven agent that navigates this manifold to assemble diverse, evidence-backed diagnostic trajectories, augmented by a linger mechanism and test-time training for case-specific calibration. Empirically, HyperWalker achieves state-of-the-art results on MIMIC-based medical report generation and the EHRXQA medical VQA benchmark, while delivering favorable inference efficiency compared with larger thinking-based models. The work demonstrates the practical potential of cross-patient, evidence-grounded diagnosis in multimodal clinical settings and points to future directions in refining temporal dynamics and scaling the hypergraph framework for broader clinical modalities.

Abstract

Automated clinical diagnosis remains a core challenge in medical AI, which usually requires models to integrate multi-modal data and reason across complex, case-specific contexts. Although recent methods have advanced medical report generation (MRG) and visual question answering (VQA) with medical vision-language models (VLMs), these methods, however, predominantly operate under a sample-isolated inference paradigm, as such processing cases independently without access to longitudinal electronic health records (EHRs) or structurally related patient examples. This paradigm limits reasoning to image-derived information alone, which ignores external complementary medical evidence for potentially more accurate diagnosis. To overcome this limitation, we propose \textbf{HyperWalker}, a \textit{Deep Diagnosis} framework that reformulates clinical reasoning via dynamic hypergraphs and test-time training. First, we construct a dynamic hypergraph, termed \textbf{iBrochure}, to model the structural heterogeneity of EHR data and implicit high-order associations among multimodal clinical information. Within this hypergraph, a reinforcement learning agent, \textbf{Walker}, navigates to and identifies optimal diagnostic paths. To ensure comprehensive coverage of diverse clinical characteristics in test samples, we incorporate a \textit{linger mechanism}, a multi-hop orthogonal retrieval strategy that iteratively selects clinically complementary neighborhood cases reflecting distinct clinical attributes. Experiments on MRG with MIMIC and medical VQA on EHRXQA demonstrate that HyperWalker achieves state-of-the-art performance. Code is available at: https://github.com/Bean-Young/HyperWalker

Figures (4)

• Figure 1: Conventional medical VLMs rely on sample-isolated inference, whereas HyperWalker performs structured multi-hop diagnostic reasoning over an implicit multimodal clinical hypergraph.
• Figure 2: Architecture of HyperWalker, illustrating (a) multimodal encoding and fusion with dynamic orthogonal multi-hop reasoning, (b) implicit clinical hypergraph iBrochure construction via heterogeneous hyperedges, and (c) the reward-guided path selector Walker for modeling the diagnosis pathway.
• Figure 3: Case Demo for MIMIC
• Figure 4: Case Study for MIMIC