Revealing the Underlying Patterns: Investigating Dataset Similarity, Performance, and Generalization

TL;DR

This work tackles the generalization challenge in segmentation by introducing dataset-image distance metrics $O^{dist}$ and $I^{dist}$ that quantify similarity between a primary crack dataset and unseen data. By projecting high-dimensional image features into a PCA-reduced space and computing pairwise distances, the authors link dataset proximity to model performance across multiple architectures, enabling model selection from candidate pools. They demonstrate that adding a small number of unlabeled or sparsely labeled images from unseen datasets can substantially improve generalization, reducing labeling costs while preserving performance. The approach is validated across crack and non-crack domains, with insights into model adaptation, behavior under distribution shift, and potential for extension to other vision tasks.

Abstract

Supervised deep learning models require significant amount of labeled data to achieve an acceptable performance on a specific task. However, when tested on unseen data, the models may not perform well. Therefore, the models need to be trained with additional and varying labeled data to improve the generalization. In this work, our goal is to understand the models, their performance and generalization. We establish image-image, dataset-dataset, and image-dataset distances to gain insights into the model's behavior. Our proposed distance metric when combined with model performance can help in selecting an appropriate model/architecture from a pool of candidate architectures. We have shown that the generalization of these models can be improved by only adding a small number of unseen images (say 1, 3 or 7) into the training set. Our proposed approach reduces training and annotation costs while providing an estimate of model performance on unseen data in dynamic environments.

Figures (32)

• Figure 1: The figure shows the distance computation process discussed in the section \ref{['distancecomp']}. FE is the feature extractor/model from which high dimensional feature vectors $P_H$ and $S_H$ are obtained. The DR (dimensionality reduction) using PCA results in low dimensional representations $P_L$ and $S_L$ from q to z dimensions of the (n+m) images. Finally, the $I^{dist}$ and $O^{dist}$ are computed using the paiwise distance matrix shown in the DC(distance computation) block.
• Figure 2: Shows (a) original image, (b) DC , (c) UNet$^{++}$ and (d) ADSAM outputs on the images from $S_6$. DC model focuses on specific features like edges.
• Figure 3: Shows the comparison of the violin plots (using \ref{['plt']}) of $I^{dist}$ distribution of all the secondary datasets $S$ from $P$. The corresponding quantitative results can be seen in \ref{['distall']}.
• Figure 4: The figure shows the F-score vs $I^{dist}(S, P)$ plot for $S$. The results are computed using the DC model which was trained on $P + n, n\in\{0,1,3,7\}$ where n images are selected from each $S$.
• Figure 5: This figure shows the F-score vs $I^{dist}(S, P)$ plot for $S$. The results are computed using UNet$^{++}$ model which was trained on $P + n, n\in\{0,1,3,7\}$ where n images are selected from each $S$.