HookMIL: Revisiting Context Modeling in Multiple Instance Learning for Computational Pathology

TL;DR

HookMIL tackles the context-aggregation bottleneck in MIL for whole-slide pathology images by introducing a small set of learnable hook tokens that perform bidirectional cross-attention with instance patches, followed by inter-hook communication and instance-to-hook feedback. This design enforces a structured, low-rank representation of inter-patch dependencies with linear complexity in the number of patches, and is augmented by a Hook Diversity Loss to promote specialization across hooks. The framework can initialize hooks from various multimodal priors (visual, text, or image-ST) to inject domain knowledge, accelerating convergence and improving representation quality. Empirical results on four public pathology datasets show state-of-the-art performance with improved efficiency and interpretability, and ablations confirm the benefits of diversity regularization and multimodal initialization for robust, scalable WSI analysis.

Abstract

Multiple Instance Learning (MIL) has enabled weakly supervised analysis of whole-slide images (WSIs) in computational pathology. However, traditional MIL approaches often lose crucial contextual information, while transformer-based variants, though more expressive, suffer from quadratic complexity and redundant computations. To address these limitations, we propose HookMIL, a context-aware and computationally efficient MIL framework that leverages compact, learnable hook tokens for structured contextual aggregation. These tokens can be initialized from (i) key-patch visual features, (ii) text embeddings from vision-language pathology models, and (iii) spatially grounded features from spatial transcriptomics-vision models. This multimodal initialization enables Hook Tokens to incorporate rich textual and spatial priors, accelerating convergence and enhancing representation quality. During training, Hook tokens interact with instances through bidirectional attention with linear complexity. To further promote specialization, we introduce a Hook Diversity Loss that encourages each token to focus on distinct histopathological patterns. Additionally, a hook-to-hook communication mechanism refines contextual interactions while minimizing redundancy. Extensive experiments on four public pathology datasets demonstrate that HookMIL achieves state-of-the-art performance, with improved computational efficiency and interpretability. Codes are available at https://github.com/lingxitong/HookMIL.

Figures (6)

• Figure 1: Comparison of different context modeling paradigms in MIL: (a) non-interactive context modeling, (b) self-interactive context modeling, and (c) hook-interactive context modeling.
• Figure 2: Overall framework of the proposed HookMIL for WSI analysis, including patch feature extraction, hook initialization, hook casting, and hook aggregation.
• Figure 3: Text-based initialization of hook tokens using prostatic grading architectural patterns.
• Figure 4: Attention maps of different hook tokens under varying Hook Diversity Loss weights.
• Figure 5: Hyperparameter ablation of HookMIL. (a) Ablation of Hook Diversity Loss. (b) Ablation of Hook Token Numbers.

Theorems & Definitions (9)

• Proposition 1: Rank-$K$ attention operator
• proof
• Remark 1: Diversity and basis incoherence
• Lemma 1: Gradient to instances
• proof
• Lemma 2: Gradient to hooks
• proof
• Theorem 1: Bidirectional connectivity with path length $\le 2$
• proof