Calibration of Network Confidence for Unsupervised Domain Adaptation Using Estimated Accuracy
Coby Penso, Jacob Goldberger
TL;DR
This work tackles the problem of calibrating neural network confidence when transferring a model from a labeled source domain to an unlabeled target domain. It introduces Unsupervised Target Domain Calibration (UTDC), which estimates the target-domain accuracy from unlabeled target data and calibrates directly on target samples by minimizing the adaECE measure across confidence bins. A key idea is to rescale target-bin accuracies by the observed source-target accuracy ratio, enabling realistic target calibration without target labels, and to apply calibration methods such as Temperature Scaling via grid search. Across four standard domain-adaptation benchmarks, UTDC outperforms importance-weighting based calibrators and approaches oracle-target calibration, demonstrating robustness to the accuracy-estimation method and offering a practical standard for target-domain confidence calibration.
Abstract
This study addresses the problem of calibrating network confidence while adapting a model that was originally trained on a source domain to a target domain using unlabeled samples from the target domain. The absence of labels from the target domain makes it impossible to directly calibrate the adapted network on the target domain. To tackle this challenge, we introduce a calibration procedure that relies on estimating the network's accuracy on the target domain. The network accuracy is first computed on the labeled source data and then is modified to represent the actual accuracy of the model on the target domain. The proposed algorithm calibrates the prediction confidence directly in the target domain by minimizing the disparity between the estimated accuracy and the computed confidence. The experimental results show that our method significantly outperforms existing methods, which rely on importance weighting, across several standard datasets.