On the Bayes Inconsistency of Disagreement Discrepancy Surrogates

TL;DR

The paper tackles distribution shift by scrutinizing disagreement-discrepancy surrogates and revealing that common surrogates are not Bayes consistent. It introduces a principled decomposition and proves inconsistency for existing methods, then presents a new cross-entropy-based surrogate with a novel disagreement loss that is provably Bayes consistent for maximizing disagreement discrepancy. The authors validate the approach with extensive experiments on covariate-shift bounds and harmful-shift detection, showing improved estimation, robustness to adversarial targets, and higher statistical power. This work provides a solid theoretical foundation for reliable use of disagreement discrepancy in robustness analysis and shift-detection tasks, with practical benefits for real-world deployment under distribution shift.

Abstract

Deep neural networks often fail when deployed in real-world contexts due to distribution shift, a critical barrier to building safe and reliable systems. An emerging approach to address this problem relies on \emph{disagreement discrepancy} -- a measure of how the disagreement between two models changes under a shifting distribution. The process of maximizing this measure has seen applications in bounding error under shifts, testing for harmful shifts, and training more robust models. However, this optimization involves the non-differentiable zero-one loss, necessitating the use of practical surrogate losses. We prove that existing surrogates for disagreement discrepancy are not Bayes consistent, revealing a fundamental flaw: maximizing these surrogates can fail to maximize the true disagreement discrepancy. To address this, we introduce new theoretical results providing both upper and lower bounds on the optimality gap for such surrogates. Guided by this theory, we propose a novel disagreement loss that, when paired with cross-entropy, yields a provably consistent surrogate for disagreement discrepancy. Empirical evaluations across diverse benchmarks demonstrate that our method provides more accurate and robust estimates of disagreement discrepancy than existing approaches, particularly under challenging adversarial conditions.

Figures (7)

• Figure 1: Comparison of disagreement discrepancy estimates for each surrogate. Left: Estimated vs. maximum achieved discrepancy across 130 shifts/models, where proximity to dashed line indicates better performance. Right: Frequency of achieving maximum discrepancy.
• Figure 2: Calibration of error bounds: observed vs. desired violation rates ($\delta$). The dashed $y=x$ line represents perfect calibration. Our surrogate demonstrates improved calibration.
• Figure 3: Comparison of disagreement discrepancy estimates for each surrogate under adversarial attacks on target data. Top: Estimated vs. maximum achieved disagreement discrepancy for each surrogate (GLK23, Ours, RG23), faceted by fraction of attacked instances. Points closer to the dashed line indicate better performance. Bottom: Corresponding bar plots displaying the rate at which each surrogate achieves rank 1 (highest), 2, or 3 (lowest) disagreement discrepancy.
• Figure 4: ROC curves for harmful shift detection on UCI-HD. Error bars indicate 95% bootstrapped confidence intervals.
• Figure 5: Comparison of error bounds versus actual error on target data for our surrogate and that of rosenfeld2023provable. Each point represents a shift model, with points above the dashed line indicating bound violations. Results are disaggregated by training method: non-domain adversarial training (left) versus domain-adversarial training (right).

Theorems & Definitions (33)

• definition 1: Disagreement Discrepancy
• definition 2: Surrogate Disagreement Discrepancy
• definition 3: Bayes consistency for disagreement discrepancy
• theorem 1: store=thm:rg-glk-lb-convex, label=thm:rg-glk-lb-convex
• corollary 1: store=cor:rg23-inconsistent, label=cor:rg23-inconsistent
• theorem 2: store=thm:ours-ub-concave, label=thm:ours-ub-concave
• corollary 2: store=cor:ours-consistent, label=cor:ours-consistent
• remark 1
• definition 4
• proposition 1