Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-shot Algorithm Assurance

Dang Nguyen, Sunil Gupta

TL;DR

This work tackles the problem of Model Assurance under Image Distortion (MAID) by predicting, for a given distortion level, whether a pre-trained image classifier maintains accuracy above a threshold. It introduces a Gaussian-process Level Set Estimation (LSE) framework with Straddle-based active sampling to efficiently identify distortion regions where the model remains usable, and extends to few-shot scenarios by augmenting data with synthetic images generated by a CVAE. The CVAE is enhanced with a distribution loss that matches low-level feature statistics from the target model and a prediction loss to ensure realism and recognizability of generated images, followed by a post-processing step to select high-confidence samples. Across MNIST, Fashion, CIFAR-10/100, and Tiny-ImageNet, the proposed LSE-C and FS-LSE-C methods consistently outperform baselines, demonstrating strong practical value for deploying classifiers under varied distortions and in data-scarce domains.

Abstract

In image classification tasks, deep learning models are vulnerable to image distortion. For successful deployment, it is important to identify distortion levels under which the model is usable i.e. its accuracy stays above a stipulated threshold. We refer to this problem as Model Assurance under Image Distortion, and formulate it as a classification task. Given a distortion level, our goal is to predict if the model's accuracy on the set of distorted images is greater than a threshold. We propose a novel classifier based on a Level Set Estimation (LSE) algorithm, which uses the LSE's mean and variance functions to form the classification rule. We further extend our method to a "few sample" setting where we can only acquire few real images to perform the model assurance process. Our idea is to generate extra synthetic images using a novel Conditional Variational Autoencoder model with two new loss functions. We conduct extensive experiments to show that our classification method significantly outperforms strong baselines on five benchmark image datasets.

Few-shot Algorithm Assurance

TL;DR

This work tackles the problem of Model Assurance under Image Distortion (MAID) by predicting, for a given distortion level, whether a pre-trained image classifier maintains accuracy above a threshold. It introduces a Gaussian-process Level Set Estimation (LSE) framework with Straddle-based active sampling to efficiently identify distortion regions where the model remains usable, and extends to few-shot scenarios by augmenting data with synthetic images generated by a CVAE. The CVAE is enhanced with a distribution loss that matches low-level feature statistics from the target model and a prediction loss to ensure realism and recognizability of generated images, followed by a post-processing step to select high-confidence samples. Across MNIST, Fashion, CIFAR-10/100, and Tiny-ImageNet, the proposed LSE-C and FS-LSE-C methods consistently outperform baselines, demonstrating strong practical value for deploying classifiers under varied distortions and in data-scarce domains.

Abstract

In image classification tasks, deep learning models are vulnerable to image distortion. For successful deployment, it is important to identify distortion levels under which the model is usable i.e. its accuracy stays above a stipulated threshold. We refer to this problem as Model Assurance under Image Distortion, and formulate it as a classification task. Given a distortion level, our goal is to predict if the model's accuracy on the set of distorted images is greater than a threshold. We propose a novel classifier based on a Level Set Estimation (LSE) algorithm, which uses the LSE's mean and variance functions to form the classification rule. We further extend our method to a "few sample" setting where we can only acquire few real images to perform the model assurance process. Our idea is to generate extra synthetic images using a novel Conditional Variational Autoencoder model with two new loss functions. We conduct extensive experiments to show that our classification method significantly outperforms strong baselines on five benchmark image datasets.
Paper Structure (27 sections, 8 equations, 4 figures, 7 tables)

This paper contains 27 sections, 8 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Background
  3. The Proposed Framework
  4. Model assurance under image distortion
  5. Model assurance with few images
  6. Generating synthetic images
  7. Training our LSE-based classifier
  8. Experiments and Discussions
  9. Experiment settings
  10. Datasets
  11. Constructing the test set for the MAID problem
  12. Evaluation metric
  13. Training the image classifier $T$
  14. Baselines
  15. Results on MNIST and Fashion
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Model misclassifies images rotated by $20^{\circ}$.
  • Figure 2: Synthetic images generated by CVAE and our generative model with different losses. "PRED loss" and "DIST loss" mean our model only uses the prediction loss (Eq. (\ref{['eq:pred-loss']})) or the distribution loss (Eq. (\ref{['eq:dist-loss']})). The text on the top indicates the labels of generated images.
  • Figure 3: F1-score vs. confidence level $\alpha$ on CIFAR-10.
  • Figure 4: "Positive" and "negative" points classified by our methods vs. the ground-truth on CIFAR-10. Red dots denote "positive" points while blue dots denote "negative" points.