Patch2Loc: Learning to Localize Patches for Unsupervised Brain Lesion Detection

Abstract

Detecting brain lesions as abnormalities observed in magnetic resonance imaging (MRI) is essential for diagnosis and treatment. In the search of abnormalities, such as tumors and malformations, radiologists may benefit from computer-aided diagnostics that use computer vision systems trained with machine learning to segment normal tissue from abnormal brain tissue. While supervised learning methods require annotated lesions, we propose a new unsupervised approach (Patch2Loc) that learns from normal patches taken from structural MRI. We train a neural network model to map a patch back to its spatial location within a slice of the brain volume. During inference, abnormal patches are detected by the relatively higher error and/or variance of the location prediction. This generates a heatmap that can be integrated into pixel-wise methods to achieve finer-grained segmentation. We demonstrate the ability of our model to segment abnormal brain tissues by applying our approach to the detection of tumor tissues in MRI on T2-weighted images from BraTS2021 and MSLUB datasets and T1-weighted images from ATLAS and WMH datasets. We show that it outperforms the state-of-the art in unsupervised segmentation. The implementation for this work can be found on our \href{https://github.com/bakerhassan/Patch2Loc}{GitHub page}. This paper has been accepted at AISTATS 2026.

Figures (11)

• Figure 1: Schematic diagram for Patch2Loc. (Top) Training stage: A randomly selected patch from a normal MRI slice is paired with its two-dimensional location $Y$ and slice index $A$. The relationship between the image patch, slice, and its location is modeled as a conditional distribution $P_\theta(Y|X,A)$ using a 2D Gaussian distribution defined by the mean and the variance of each coordinate that are functions of the patch and slice index. Note that patches with less than $20\%$ of its content as brain tissues, as in the yellow patch, are rejected. (Bottom) Inference stage: Overlapping patches with a fixed stride are extracted as in convolution from an MRI slice and the model is applied to each patch. The squared norm of the error between the model's predicted mean and the patch's true location creates an error map. Likewise, the sum of the variances create a variance map. Together the sum of the logarithms of the errors and variances highlight anomalous areas.
• Figure 2: Visualization of Patch2Loc's output (blue ellipses) for patches captured from the same location (red dot) across different subjects in the BraTS dataset, overlaid on a representative T1-weighted registered slice (without abnormalities). (Top row) Predictions of normal patches. (Bottom row) Corresponding predictions of abnormal patches. The predicted Gaussian distribution is visualized as an ellipse, where the center represents the predicted mean, and the major and minor axes correspond to two standard deviations.
• Figure 3: From left to right: 1D KDE for log-error, and log-variance, 2D KDE for log-error and log-variance, and 1D KDE for the abnormality score for normal patches (blue) and abnormal patches (red) for the BraTS dataset.
• Figure 4: Visualization of our model's anomaly detection performance on pathological slices from the BraTS, ATLAS, MSLUB, and WMH datasets, the model's heatmap successfully localizes abnormalities, showing high correlation with the ground truth masks (white). Colormap ranges from 0 (blue) to 12 (red).
• Figure 5: Visualization of our model's heatmap on healthy control slices (T1, T2, T1, T2, across the rows) from the IXI dataset testing set. The model correctly produces low, diffuse activations. Colormap ranges from 0 (blue) to 12 (red).