Diffusion Representations for Fine-Grained Image Classification: A Marine Plankton Case Study

TL;DR

It is shown that a frozen diffusion backbone enables strong fine-grained recognition by probing intermediate denoising features across layers and timesteps and training a linear classifier for each pair in a real-world plankton-monitoring setting with practical impact.

Abstract

Diffusion models have emerged as state-of-the-art generative methods for image synthesis, yet their potential as general-purpose feature encoders remains underexplored. Trained for denoising and generation without labels, they can be interpreted as self-supervised learners that capture both low- and high-level structure. We show that a frozen diffusion backbone enables strong fine-grained recognition by probing intermediate denoising features across layers and timesteps and training a linear classifier for each pair. We evaluate this in a real-world plankton-monitoring setting with practical impact, using controlled and comparable training setups against established supervised and self-supervised baselines. Frozen diffusion features are competitive with supervised baselines and outperform other self-supervised methods in both balanced and naturally long-tailed settings. Out-of-distribution evaluations on temporally and geographically shifted plankton datasets further show that frozen diffusion features maintain strong accuracy and Macro F1 under substantial distribution shift.

Figures (6)

• Figure 1: Principal Component Analysis (PCA) visualization of diffusion features. Using random samples, we compute PCA on the diffusion features and map the first three principal components to the RGB channels. For each sample, we show the clean target image and its noisy versions at timesteps $t \in \{25, 200, 600\}$ with the corresponding PCA colorization overlaid on the noisy image. Background regions are suppressed by masking pixels with low values on the first principal component. This visualization is inspired by oquab2024dinov2learningrobustvisual.
• Figure 2: Linear probe accuracy across decoder readout locations $\ell \in \{1,\dots,12\}$ at the optimal noise level $t^\star = 25$. The DDPM runs for $T = 1000$ total timesteps, and we probe the subset $\{1, 10, 25, 50, 75, 100, 200, 400, 600\}$.
• Figure 3: Visualization of learned representations via t-SNE with $k$-means (15 out of the 70 classes), with clusters matched to labels using the Hungarian algorithmkuhn1955hungarian. Mismatches represent points of the selected labels that did not fall under the correct cluster.
• Figure 4: Effect of loss on linear probe accuracy across noise. Linear probe accuracy as a function of $\log \operatorname{SNR}(t)$ for the best decoder readout $(t^\star, \ell^\star)$. We compare two denoisers trained with identical architecture and schedule but different timestep weightings: uniform MSE weighting $w_{\text{MSE}}(t) \equiv 1$ and MinSNR ($\gamma$=5) weighting $w_{\text{MinSNR}}(t)$. High SNR corresponds to nearly clean inputs, low SNR to heavily noised inputs. Balanced dataset.
• Figure 5: Training and validation loss (left axis) and Fréchet Inception Distance (FID, right axis) on the validation set as a function of epoch for the diffusion model. FID is computed from 10k generated samples compared to the validation set, with grayscale ROIs replicated to three channels for Inception-V3.