Sensitivity analysis of image classification models using generalized polynomial chaos

TL;DR

This paper addresses the challenge of uncertainty in image-classification models under domain shifts by applying sensitivity analysis (SA) using generalized polynomial chaos (GPC). It builds a surrogate model of the transformed input-output behavior via a GPC expansion, enabling efficient computation of Sobol indices $S_\tau$ to quantify the influence of perturbations (e.g., brightness, rotation, tilt) and their interactions on predictions. The approach is demonstrated on two manufacturing-relevant case studies: TIG welding seam classification with a fine-tuned ResNet18 and a BMW emblem detection model, revealing that geometric perturbations (tilt/rotation) and their interactions can significantly impact performance, while brightness effects are often smaller. The work provides a framework to diagnose robustness and guide improvements under distributional shifts, with potential to enhance reliability in production-quality predictions.

Abstract

Integrating advanced communication protocols in production has accelerated the adoption of data-driven predictive quality methods, notably machine learning (ML) models. However, ML models in image classification often face significant uncertainties arising from model, data, and domain shifts. These uncertainties lead to overconfidence in the classification model's output. To better understand these models, sensitivity analysis can help to analyze the relative influence of input parameters on the output. This work investigates the sensitivity of image classification models used for predictive quality. We propose modeling the distributional domain shifts of inputs with random variables and quantifying their impact on the model's outputs using Sobol indices computed via generalized polynomial chaos (GPC). This approach is validated through a case study involving a welding defect classification problem, utilizing a fine-tuned ResNet18 model and an emblem classification model used in BMW Group production facilities.

Figures (3)

• Figure 1: The fixed input $\boldsymbol{x}$, given the perturbation $\boldsymbol{\xi}$, is transformed using the transformation function $T(\boldsymbol{x}, \boldsymbol{\xi})$ and passed through the black-box model $f$ to obtain the probability distribution $p_Y(\boldsymbol{y} \mid \boldsymbol{\xi})$. GPC approximates a surrogate model of $f(T(\boldsymbol{x}, \boldsymbol{\xi}))$ in terms of a series expansion of multivariate orthogonal polynomials. The surrogate model is then used to compute the Sobol indices $S_\tau$.
• Figure 2: Figure (A) presents a surface plot of the GPC approximation, illustrating the relationship between brightness and rotation for the prediction probability, with tilt held constant at $0$. Figure (B) displays the approximation of brightness while both rotation and tilt are fixed at $0$.
• Figure 3: Figure (A) presents a surface plot of the GPC approximation, illustrating the relationship between brightness and rotation for the prediction probability. Figure (B) shows the experimental setup used to obtain the validation images.