Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification

Daniel Beckmann, Benjamin Risse

Abstract

Expert eye movements provide a rich, passive source of domain knowledge in radiology, offering a powerful cue for integrating diagnostic reasoning into computer-aided analysis. However, direct integration into CNN-based systems, which historically have dominated the medical image analysis domain, is challenging: gaze recordings are sequential, temporally dense yet spatially sparse, noisy, and variable across experts. As a consequence, most existing image-based models utilize reduced representations such as heatmaps. In contrast, gaze naturally aligns with transformer architectures, as both are sequential in nature and rely on attention to highlight relevant input regions. In this work, we introduce FixationFormer, a transformer-based architecture that represents expert gaze trajectories as sequences of tokens, thereby preserving their temporal and spatial structure. By modeling gaze sequences jointly with image features, our approach addresses sparsity and variability in gaze data while enabling a more direct and fine-grained integration of expert diagnostic cues through explicit cross-attention between the image and gaze token sequences. We evaluate our method on three publicly available benchmark chest X-ray datasets and demonstrate that it achieves state-of-the-art classification performance, highlighting the value of representing gaze as a sequence in transformer-based medical image analysis.

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification

Abstract

Expert eye movements provide a rich, passive source of domain knowledge in radiology, offering a powerful cue for integrating diagnostic reasoning into computer-aided analysis. However, direct integration into CNN-based systems, which historically have dominated the medical image analysis domain, is challenging: gaze recordings are sequential, temporally dense yet spatially sparse, noisy, and variable across experts. As a consequence, most existing image-based models utilize reduced representations such as heatmaps. In contrast, gaze naturally aligns with transformer architectures, as both are sequential in nature and rely on attention to highlight relevant input regions. In this work, we introduce FixationFormer, a transformer-based architecture that represents expert gaze trajectories as sequences of tokens, thereby preserving their temporal and spatial structure. By modeling gaze sequences jointly with image features, our approach addresses sparsity and variability in gaze data while enabling a more direct and fine-grained integration of expert diagnostic cues through explicit cross-attention between the image and gaze token sequences. We evaluate our method on three publicly available benchmark chest X-ray datasets and demonstrate that it achieves state-of-the-art classification performance, highlighting the value of representing gaze as a sequence in transformer-based medical image analysis.
Paper Structure (18 sections, 1 equation, 4 figures, 4 tables)

This paper contains 18 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Image Encoder
  5. Gaze Representation
  6. Integration with Image Features
  7. Image-to-Gaze Cross-Attention
  8. Two-Way Attention
  9. Experiments
  10. Datasets
  11. Training Details
  12. Results
  13. Classification Performance
  14. CXR-Gaze Dataset
  15. SIIM-ACR Dataset
  16. ...and 3 more sections

Figures (4)

  • Figure 1: FixationFormer overview: Image and gaze are encoded into separate token sequences. To infuse the image features with gaze information, we use cross-attention in one or optionally both directions throughout a stack of Transformer layers. Finally, the [CLS] token from the image encoder is passed to the prediction head.
  • Figure 2: Gaze Representation: The fixation trajectory (containing the start time, duration and spatial location for each fixation) is transformed into a token sequence. Spatial location and duration are projected using a learned linear layer, while the start times are used to encode the relative positional information using the positional embeddings proposed by vaswani2017attentionneed.
  • Figure 3: Image-to-Gaze Cross-Attention: Images and gaze features are fused by cross-attention. Output features of the attention mechanism are based on gaze features, but the skip-connection allows for conservation of image information in the output tokens. No masking is applied to allow access to all gaze tokens for each image patch.
  • Figure 4: GradCAM-based attention visualization: Comparison of model attention for correctly classified samples. Left: GradCAM image for the Image Only model. Center: Visualization of the expert fixation trajectory. Right: GradCAM image for the Cross-Attention model.