Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics

TL;DR

The paper tackles sample-size-induced bias in confusion-matrix metrics by analyzing the discrete, combinatorial structure of confusion matrices and showing how small $n$ yields metric jaggedness and undefined cases. It introduces two remedies: the MATCH Test, which assesses whether a group’s metric score is statistically consistent with a reference distribution, and Cross-Prior Smoothing (CPS), a Dirichlet-like prior-based correction that smooths a target group's confusion matrix using information from a reference group. Theoretical results quantify the distributional properties of binomial, marginal-benefit, and joint-ratio metrics and demonstrate that CPS substantially reduces metric error across 15 metrics on fairness datasets (COMPAS and Folktable), with empirically favorable bias-variance trade-offs. Overall, the work provides practical tools and open-source resources to improve the reliability and fairness of classification evaluations under small-sample or imbalanced scenarios.

Abstract

Evaluating machine learning models is crucial not only for determining their technical accuracy but also for assessing their potential societal implications. While the potential for low-sample-size bias in algorithms is well known, we demonstrate the significance of sample-size bias induced by combinatorics in classification metrics. This revelation challenges the efficacy of these metrics in assessing bias with high resolution, especially when comparing groups of disparate sizes, which frequently arise in social applications. We provide analyses of the bias that appears in several commonly applied metrics and propose a model-agnostic assessment and correction technique. Additionally, we analyze counts of undefined cases in metric calculations, which can lead to misleading evaluations if improperly handled. This work illuminates the previously unrecognized challenge of combinatorics and probability in standard evaluation practices and thereby advances approaches for performing fair and trustworthy classification methods.

Figures (7)

• Figure 1: Variability of Positive Predictive Rate for Wealth Classification Among Multiracial Individuals using the \ref{['sec:folktablesIncomeData']}: A Monte Carlo Simulation Study Based on Sample Size.
• Figure 2: Distribution Shift in Classification: Analyzing Metric Sensitivity to Sample Size with Empirical Cumulative Distribution Functions.
• Figure 3: Sample-Size-Induced Bias Among all Groups: Impact of Cross-Prior Smoothing.
• Figure 4: Comparing Cross-Prior Smoothing with the original metric, and the add-one technique.
• Figure 5: Effect of smoothing techniques on metrics with undefined values. Bashful smoothing ($\varepsilon=1e^{-10}$) reduces errors compared to no smoothing ($\varepsilon=0$). Cross-Prior Smoothing (CPS) offers further improvements, with the stronger prior ($\lambda=20$) outperforming the weaker prior ($\lambda=5$), indicating that CPS uses sufficiently informative priors.

Theorems & Definitions (31)

• Definition 3.1
• Theorem 5.1
• proof
• Theorem 5.2
• proof
• Theorem 5.3
• Theorem 6.1
• Corollary 6.1.1
• Theorem A.1
• proof