On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification

TL;DR

This paper tackles high-resolution image classification when the region of interest is tiny and memory limits preclude processing all patches. It advances IPS by embedding it in a memory-efficient cross-attention transformer and introducing a Megapixel MNIST-based testbed with fixed canvas and a controllable object-to-image ratio, plus a Bézier-curve noise generator to simulate adverse conditions. Key contributions include demonstrating that the $O2I$ threshold for generalization is modulated by training data size and that patch size tuning relative to the ROI improves generalization in low-data regimes, as well as revealing how noise resemblance to ROI impedes convergence. The work provides practical guidance for designing robust, memory-efficient patch-based classifiers in settings with extreme data scarcity and strong noise, and releases public code to foster reproducibility.

Abstract

Classifying large images with small or tiny regions of interest (ROI) is challenging due to computational and memory constraints. Weakly supervised memory-efficient patch selectors have achieved results comparable with strongly supervised methods. However, low signal-to-noise ratios and low entropy attention still cause overfitting. We explore these issues using a novel testbed on a memory-efficient cross-attention transformer with Iterative Patch Selection (IPS) as the patch selection module. Our testbed extends the megapixel MNIST benchmark to four smaller O2I (object-to-image) ratios ranging from 0.01% to 0.14% while keeping the canvas size fixed and introducing a noise generation component based on Bézier curves. Experimental results generalize the observations made on CNNs to IPS whereby the O2I threshold below which the classifier fails to generalize is affected by the training dataset size. We further observe that the magnitude of this interaction differs for each task of the Megapixel MNIST. For tasks "Maj" and "Top", the rate is at its highest, followed by tasks "Max" and "Multi" where in the latter, this rate is almost at 0. Moreover, results show that in a low data setting, tuning the patch size to be smaller relative to the ROI improves generalization, resulting in an improvement of + 15% for the megapixel MNIST and + 5% for the Swedish traffic signs dataset compared to the original object-to-patch ratios in IPS. Further outcomes indicate that the similarity between the thickness of the noise component and the digits in the megapixel MNIST gradually causes IPS to fail to generalize, contributing to previous suspicions.

Figures (10)

• Figure 1: Image "000036" from the Swedish traffic signs dataset with label "80". On the left, the original image in full resolution ($960 \times 1280$). On the right, the image is truncated to approximately $75 \times 45$ (top) and $35 \times 15$ (bottom).
• Figure 2: Visual representation of the Needle MNIST dataset (left) and the Megapixel MNIST dataset (right).
• Figure 3: Visual representation of the Megapixel MNIST dataset (150 $\times$ 150) with 10 noise digits from our method (left) and the original Megapixel MNIST (right).
• Figure 4: Experiments on megapixel MNIST with a novel noise generation component using Bézier curves that aim to resemble the number of control points found in digits. Four object-to-image ratios were tested: across four training dataset sizes. Canvas size and patch size remain fixed at $3000 \times 3000$ and $50 \times 50$, respectively, and the O2I changes by varying the digit resolutions to $28 \times 28$, $56 \times 56$, $84 \times 84$, and $112 \times 112$.
• Figure 5: Visualization of 7 noise digits with the thickness in parenthesis as well as two MNIST digits (bottom right).