Cross-Enhanced Multimodal Fusion of Eye-Tracking and Facial Features for Alzheimer's Disease Diagnosis

TL;DR

This work tackles auxiliary Alzheimer's disease diagnosis by fusing eye-tracking and facial features through a transformer-based cross-enhanced fusion framework. It introduces two novel components, the Cross-Enhanced Fusion Attention Module (CEFAM) and the Direction-Aware Convolution Module (DACM), to model inter-modal interactions and fine-grained directional facial cues, respectively. The authors also provide a synchronized multimodal dataset of 50 participants (25 AD, 25 HC) collected during a visual memory–search task. Empirical results show the proposed method surpasses single-modality and naive fusion baselines, achieving 95.11% accuracy and demonstrating the practical viability of non-invasive, cost-effective multimodal diagnostics. Collectively, the approach offers robust cross-modal representations and a scalable framework for assistive cognitive impairment screening.

Abstract

Accurate diagnosis of Alzheimer's disease (AD) is essential for enabling timely intervention and slowing disease progression. Multimodal diagnostic approaches offer considerable promise by integrating complementary information across behavioral and perceptual domains. Eye-tracking and facial features, in particular, are important indicators of cognitive function, reflecting attentional distribution and neurocognitive state. However, few studies have explored their joint integration for auxiliary AD diagnosis. In this study, we propose a multimodal cross-enhanced fusion framework that synergistically leverages eye-tracking and facial features for AD detection. The framework incorporates two key modules: (a) a Cross-Enhanced Fusion Attention Module (CEFAM), which models inter-modal interactions through cross-attention and global enhancement, and (b) a Direction-Aware Convolution Module (DACM), which captures fine-grained directional facial features via horizontal-vertical receptive fields. Together, these modules enable adaptive and discriminative multimodal representation learning. To support this work, we constructed a synchronized multimodal dataset, including 25 patients with AD and 25 healthy controls (HC), by recording aligned facial video and eye-tracking sequences during a visual memory-search paradigm, providing an ecologically valid resource for evaluating integration strategies. Extensive experiments on this dataset demonstrate that our framework outperforms traditional late fusion and feature concatenation methods, achieving a classification accuracy of 95.11% in distinguishing AD from HC, highlighting superior robustness and diagnostic performance by explicitly modeling inter-modal dependencies and modality-specific contributions.

Figures (8)

• Figure 1: Visual memory and search paradigm.
• Figure 2: Multimodal data acquisition scenario.
• Figure 3: Multimodal cross-enhanced fusion network.
• Figure 4: Direction-aware convolution module.
• Figure 5: Cross-enhanced fusion attention module.