Towards Calibration Enhanced Network by Inverse Adversarial Attack

TL;DR

This work tackles miscalibration in OCR-based HMI validation by introducing Inverse Adversarial Attack (IAA), which generates underconfident yet correctly classified examples, and Inverse Adversarial Training (IAT) to defend against such underconfidence. The authors design a two-term cross-entropy loss for IAA and derive its optimization properties, showing how the ground-truth probability can be driven toward underconfidence under controlled perturbations. Through extensive experiments on CIFAR-10 and qualitative NeurIPS data, they demonstrate that IAA effectively reveals underconfidence, and that integrating IAEs via IAT, especially when combined with standard adversarial training (AT), yields models that are both robust and well-calibrated. The work also discusses transferability, showing strong whitebox effects but limited cross-model transfer, and suggests future directions for enhancing cross-architecture applicability.

Abstract

Test automation has become increasingly important as the complexity of both design and content in Human Machine Interface (HMI) software continues to grow. Current standard practice uses Optical Character Recognition (OCR) techniques to automatically extract textual information from HMI screens for validation. At present, one of the key challenges faced during the automation of HMI screen validation is the noise handling for the OCR models. In this paper, we propose to utilize adversarial training techniques to enhance OCR models in HMI testing scenarios. More specifically, we design a new adversarial attack objective for OCR models to discover the decision boundaries in the context of HMI testing. We then adopt adversarial training to optimize the decision boundaries towards a more robust and accurate OCR model. In addition, we also built an HMI screen dataset based on real-world requirements and applied multiple types of perturbation onto the clean HMI dataset to provide a more complete coverage for the potential scenarios. We conduct experiments to demonstrate how using adversarial training techniques yields more robust OCR models against various kinds of noises, while still maintaining high OCR model accuracy. Further experiments even demonstrate that the adversarial training models exhibit a certain degree of robustness against perturbations from other patterns.

Figures (8)

• Figure 1: An image of a Samoyed dog from the ImageNet dataset, predicted by a pretrained ResNet50 DNN classifier, demonstrates the presence of the Inverse Adversarial Example. (a) The clear input is correctly classified as Samoyed with a confidence of $40.1\%$. (b) The Adversarial Example, with a perturbation of $\epsilon=0.03$, is misclassified as Eskimo with a confidence of $76\%$. (c) The Inverse Adversarial Example, also with a perturbation of $\epsilon=0.03$, is correctly classified as Samoyed, but with a very low confidence of $2.3\%$. The prediction confidence distributions for the corresponding inputs are shown at the bottom.
• Figure 2: The analysis of $\mathcal{L}$ in \ref{['IAA:eq:optim']} with $\lambda=5,~K=10$. $\mathcal{L}$ reaches its minimum when $y_G = 0.25$.
• Figure 3: The framework of Inverse Adversarial Training.
• Figure 4: Apply different IAA with different $\lambda$ against to a well-trained model on CIFAR10. ACC, CONF and MCS are shown along with $\epsilon$ ranging from $0.02$ to $0.1$.
• Figure 5: MCS of the examples generated by uniform noise (UN), PGD attack (AA), IAA against to PT model with maximum perturbation $\epsilon$ ranging from $0.02$ to $0.1$. The red dot line indicates the perfect calibration, i.e., MCS=0.