Test-time Adaptation Meets Image Enhancement: Improving Accuracy via Uncertainty-aware Logit Switching

TL;DR

This work hypothesizes that enhancing the input image reduces prediction’s uncertainty and increase the accuracy of TTA methods and proposes a novel method: Test-time Enhancer and Classifier Adaptation (TECA), which reduces prediction’s uncertainty and increases accuracy of TTA methods despite having no hyperparameters and little parameter overhead.

Abstract

Deep neural networks have achieved remarkable success in a variety of computer vision applications. However, there is a problem of degrading accuracy when the data distribution shifts between training and testing. As a solution of this problem, Test-time Adaptation~(TTA) has been well studied because of its practicality. Although TTA methods increase accuracy under distribution shift by updating the model at test time, using high-uncertainty predictions is known to degrade accuracy. Since the input image is the root of the distribution shift, we incorporate a new perspective on enhancing the input image into TTA methods to reduce the prediction's uncertainty. We hypothesize that enhancing the input image reduces prediction's uncertainty and increase the accuracy of TTA methods. On the basis of our hypothesis, we propose a novel method: Test-time Enhancer and Classifier Adaptation~(TECA). In TECA, the classification model is combined with the image enhancement model that transforms input images into recognition-friendly ones, and these models are updated by existing TTA methods. Furthermore, we found that the prediction from the enhanced image does not always have lower uncertainty than the prediction from the original image. Thus, we propose logit switching, which compares the uncertainty measure of these predictions and outputs the lower one. In our experiments, we evaluate TECA with various TTA methods and show that TECA reduces prediction's uncertainty and increases accuracy of TTA methods despite having no hyperparameters and little parameter overhead.

Figures (4)

• Figure 1: Preliminary experiments on our hypothesis. We updated ResNeXt resnext with Tent on the CIFAR-100-C dataset and measured error rates and entropy. We tested clean images and ones corrupted at five severity levels, where higher severity indicates stronger corruption and lower image quality, while lower severity indicates cleaner images and higher image quality.
• Figure 2: Overview of TECA. The image enhancement model transforms the target image into a recognition-friendly one. The classification model predicts both original and recognition-friendly images. TECA compares the uncertainty of the predictions and switches predictions to the lower one. The model is updated from the prediction using the TTA methods. At this time, the gradient of the image enhancement model is rescaled to align the parameter update speed of each model that stabilizes the updates. Additionally, the BN statistics of the image enhancement model are frozen to preserve source knowledge.
• Figure 3: Gaussian noise at severity level 5 images and URIE urie-enhanced images and their confidence score in the ImageNet-C dataset. The top and bottom rows are the original and enhanced images, respectively.
• Figure 4: Trade-off between error rate and number of parameters for the standard CTTA task. We compared the TTA method with TECA using ResNet-18, 34, 50, 101, and 152 as classification models. The blue and orange lines show TTA method without and with TECA.