Misaligned by Design: Incentive Failures in Machine Learning

TL;DR

The paper addresses how asymmetric, human-aligned loss functions can misalign machine learning when the agent learns under such incentives. By modeling the loss choice as an incentive-design problem with human utility $u(a,y)$, it distinguishes external alignment (predict-and-adjust for human objective) from internal alignment (loss shaping learning). The key contribution is showing that training with a utility-weighted loss can dampen the marginal value of learning, making ex post adjustments to unweighted predictions preferable in multi-class settings; this is formalized via the utility-weighted prediction $p^u(q)$ and the residual learning loss. Empirically, the authors demonstrate across pneumonia detection and CIFAR classification that Ex Post Weighting consistently outperforms Weighted Training on both the machine’s objective (weighted loss) and the downstream classification utility. The work thus shifts the practical emphasis from embedding human costs into training to designing incentives that preserve information value while allowing post hoc alignment, with implications for cost-sensitive and alignment-aware learning.

Abstract

The cost of error in many high-stakes settings is asymmetric: misdiagnosing pneumonia when absent is an inconvenience, but failing to detect it when present can be life-threatening. Because of this, artificial intelligence (AI) models used to assist such decisions are frequently trained with asymmetric loss functions that incorporate human decision-makers' trade-offs between false positives and false negatives. In two focal applications, we show that this standard alignment practice can backfire. In both cases, it would be better to train the machine learning model with a loss function that ignores the human's objective and then adjust predictions ex post according to that objective. We rationalize this result using an economic model of incentive design with endogenous information acquisition. The key insight from our theoretical framework is that machine classifiers perform not one but two incentivized tasks: choosing how to classify and learning how to classify. We show that while the adjustments engineers use correctly incentivize choosing, they can simultaneously reduce the incentives to learn. Our formal treatment of the problem reveals that methods embraced for their intuitive appeal can in fact misalign human and machine objectives in predictable ways.

Figures (9)

• Figure 1: Weighted loss in the test sample across training incentives by training epoch, averaged across five runs. In red are the average weighted losses from training with the weights that reflect the human's objective, and in blue are the average weighted losses from training without weights, but accounting for the weights by transforming predictions ex post. Dots represent the minimum training epoch and dotted lines the corresponding weighted loss.
• Figure 2: Incentives to choose and learn in the case of unweighted and weighted binary classification. Class weighting incentivizes distorting predictions as a function of posterior probabilities (left panel). This distorts the marginal benefit of learning such probabilities (center panel), lowering and distorting the overall incentives for learning (right panel).
• Figure 3: The components of the AI system.
• Figure 4: Weighted loss when emphasizing pneumonia, evaluated in test sample. The black lines represent the weighted loss (across training runs) from unweighted training without adjusting predictions. The red lines represent the weighted loss from weighted training. Finally, the blue lines represent the weighted loss from unweighted training after analytically adjusting predictions. For each run, the point denotes the minimal weighted loss and the training epoch at which it is achieved. Consistently across training runs, we outperform the machine on its own objective by not training according to downstream incentives, but rather analytically adjusting for them ex post.
• Figure 5: Classification utility when emphasizing pneumonia, evaluated in test sample. The black lines represent the achieved utility (across training runs) from unweighted training without adjusting predictions. The red lines represent utility from weighted training. Finally, the blue lines represent utility from unweighted training after analytically adjusting predictions. For each run, the point denotes the maximal classification utility and the training epoch at which it is achieved. Consistently across training runs, we outperform the machine on the downstream utility objective by not training according to downstream incentives, but rather analytically adjusting for them ex post. Based on the preceding Figure \ref{['fig:loss-pneumonia']}, we attribute this to underperformance and suppressed learning when the machine is trained according to utility-weighted cross entropy.

Theorems & Definitions (11)

• Theorem 1: Optimal Utility-Weighted Prediction
• Proposition 1: Learning Identity
• Proposition 2: Incentives Underlying Learning
• Proposition 3: saerens2002adjust
• Theorem 2: Analytical Recalibration
• Lemma 1
• proof : Proof of \ref{['thm:alignment-outputting']}
• proof : Proof of \ref{['thm:incentives-learning']}
• proof : Proof of \ref{['thm:recalibrate']}
• proof : Proof of \ref{['thm:convex-closed']}