Fourier Transform Multiple Instance Learning for Whole Slide Image Classification

TL;DR

This work addresses the challenge of modeling long-range global context in whole slide image classification with multiple instance learning. It introduces FFT-MIL, which adds a compact frequency-domain FFT-Block that processes low-frequency crops and fuses learned global frequency features with spatial patch features. The approach yields consistent improvements across six MIL baselines on BRACS, LUAD, and IMP, with macro F1 increasing by 3.51% and AUC by 1.51%. The results demonstrate that frequency-domain learning provides a practical, generalizable augmentation to MIL, enhancing scalability and accuracy in computational pathology.

Abstract

Whole Slide Image (WSI) classification relies on Multiple Instance Learning (MIL) with spatial patch features, yet existing methods struggle to capture global dependencies due to the immense size of WSIs and the local nature of patch embeddings. This limitation hinders the modeling of coarse structures essential for robust diagnostic prediction. We propose Fourier Transform Multiple Instance Learning (FFT-MIL), a framework that augments MIL with a frequency-domain branch to provide compact global context. Low-frequency crops are extracted from WSIs via the Fast Fourier Transform and processed through a modular FFT-Block composed of convolutional layers and Min-Max normalization to mitigate the high variance of frequency data. The learned global frequency feature is fused with spatial patch features through lightweight integration strategies, enabling compatibility with diverse MIL architectures. FFT-MIL was evaluated across six state-of-the-art MIL methods on three public datasets (BRACS, LUAD, and IMP). Integration of the FFT-Block improved macro F1 scores by an average of 3.51% and AUC by 1.51%, demonstrating consistent gains across architectures and datasets. These results establish frequency-domain learning as an effective and efficient mechanism for capturing global dependencies in WSI classification, complementing spatial features and advancing the scalability and accuracy of MIL-based computational pathology.

Figures (10)

• Figure 1: Overview of the proposed Fourier Transform Multiple Instance Learning (FFT-MIL) framework integrated with CLAM lu2021data for WSI classification. The FFT-Block extracts a global frequency feature from a given WSI, which is fused with the output of CLAM’s lu2021data attention backbone via addition to introduce global context at a stage where patch-level information has been aggregated. While illustrated with CLAM lu2021data, the FFT-Block is modular and can be integrated into other MIL methods in a similar fashion.
• Figure 2: Overview of our proposed preprocessing pipeline for obtaining low-frequency representations of WSIs. The CLAM lu2021data patch extraction branch (top) uses a $16\times$ to $64\times$ downsampled WSI for tissue segmentation, which is then aligned to the full-resolution image for patch extraction. The FFT-MIL branch (middle) operates on a $4\times$ downsampled WSI, applying FFT, frequency shift, and center cropping to retain low-frequency components. The reconstruction branch (bottom right), included for visualization purposes only, performs inverse FFT and padding to approximate the original image. A visual comparison of original and reconstructed patches is shown (bottom left).
• Figure 3: Architectures of our proposed FFT-Block and the FFT-Vanilla Block. FFT-Block: A modular component that operates entirely in the frequency domain using repeated 2D $3 \times 3$ convolutions, ReLU activations, and $2 \times 2$ max pooling. The 2D output is normalized via Min-Max scaling and passed to a multi-layer perceptron block, producing a global frequency feature for integration with MIL-based architectures or direct classification. FFT-Vanilla Block: A baseline component used to illustrate the role of the iFFT in current frequency-domain architectures. It applies repeated 2D $3 \times 3$ convolutions, each followed by an inverse FFT, Batch Normalization, ReLU, and max pooling. An FFT is applied after each block to return to the frequency domain before the next convolution. The final block omits the FFT to retain the spatial representation, which is passed to an MLP for the same downstream uses as the FFT-Block.
• Figure 4: Information retention versus normalized input size for patch-based and frequency-based representations. A normalized input size of $1.0$ corresponds to full-image coverage. Patch-based input reflects the number of extracted patches multiplied by channel count and embedding dimensionality. Frequency-based input reflects the area of a radial crop in the Fourier domain. As shown, frequency-based inputs retain substantially more information at lower input sizes, highlighting their data efficiency in capturing global context compared to patch-based inputs.
• Figure 5: Normalized confusion matrices comparing the classification performance of the baseline CLAM model (left) and the proposed FFT-MIL model (right) on BRACS brancati2022bracs. Each matrix illustrates the normalized distribution of true versus predicted class labels. Summary metrics below each matrix include Accuracy (Acc), Precision (Prec), Recall (Rec), F1 score (F1), and Area Under the Curve (AUC). FFT-MIL demonstrates improved predictive performance as indicated by higher diagonal values in the confusion matrix.