Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Impact of Case Correction Methods on the Fairness of COVID-19 Predictive Models

Daniel Smolyak, Saad Abrar, Naman Awasthi, Vanessa Frias-Martinez

TL;DR

The paper investigates whether COVID-19 case correction methods, designed to address under-counts, alter the fairness of county-level case predictions across racial groups. It adapts two correction approaches—Dynamics in Infection Numbers and CFR Benchmark—and evaluates their impact using quantile regression predictions and the Accuracy Equality Ratio ($AER$). Results are mixed: the dynamics-based method generally preserves or improves fairness, while the CFR-based method often degrades fairness, depending on study period and race-assignment scheme. The work highlights that correction methods can shift the burden of prediction errors toward marginalized groups, underscoring the need for auditing and careful consideration before deploying such corrections in policy-relevant decision-making.

Abstract

One of the central difficulties of addressing the COVID-19 pandemic has been accurately measuring and predicting the spread of infections. In particular, official COVID-19 case counts in the United States are under counts of actual caseloads due to the absence of universal testing policies. Researchers have proposed a variety of methods for recovering true caseloads, often through the estimation of statistical models on more reliable measures, such as death and hospitalization counts, positivity rates, and demographics. However, given the disproportionate impact of COVID-19 on marginalized racial, ethnic, and socioeconomic groups, it is important to consider potential unintended effects of case correction methods on these groups. Thus, we investigate two of these correction methods for their impact on a downstream COVID-19 case prediction task. For that purpose, we tailor an auditing approach and evaluation protocol to analyze the fairness of the COVID-19 prediction task by measuring the difference in model performance between majority-White counties and majority-minority counties. We find that one of the correction methods improves fairness, decreasing differences in performance between majority-White and majority-minority counties, while the other method increases differences, introducing bias. While these results are mixed, it is evident that correction methods have the potential to exacerbate existing biases in COVID-19 case data and in downstream prediction tasks. Researchers planning to develop or use case correction methods must be careful to consider negative effects on marginalized groups.

Assessing the Impact of Case Correction Methods on the Fairness of COVID-19 Predictive Models

TL;DR

The paper investigates whether COVID-19 case correction methods, designed to address under-counts, alter the fairness of county-level case predictions across racial groups. It adapts two correction approaches—Dynamics in Infection Numbers and CFR Benchmark—and evaluates their impact using quantile regression predictions and the Accuracy Equality Ratio (). Results are mixed: the dynamics-based method generally preserves or improves fairness, while the CFR-based method often degrades fairness, depending on study period and race-assignment scheme. The work highlights that correction methods can shift the burden of prediction errors toward marginalized groups, underscoring the need for auditing and careful consideration before deploying such corrections in policy-relevant decision-making.

Abstract

One of the central difficulties of addressing the COVID-19 pandemic has been accurately measuring and predicting the spread of infections. In particular, official COVID-19 case counts in the United States are under counts of actual caseloads due to the absence of universal testing policies. Researchers have proposed a variety of methods for recovering true caseloads, often through the estimation of statistical models on more reliable measures, such as death and hospitalization counts, positivity rates, and demographics. However, given the disproportionate impact of COVID-19 on marginalized racial, ethnic, and socioeconomic groups, it is important to consider potential unintended effects of case correction methods on these groups. Thus, we investigate two of these correction methods for their impact on a downstream COVID-19 case prediction task. For that purpose, we tailor an auditing approach and evaluation protocol to analyze the fairness of the COVID-19 prediction task by measuring the difference in model performance between majority-White counties and majority-minority counties. We find that one of the correction methods improves fairness, decreasing differences in performance between majority-White and majority-minority counties, while the other method increases differences, introducing bias. While these results are mixed, it is evident that correction methods have the potential to exacerbate existing biases in COVID-19 case data and in downstream prediction tasks. Researchers planning to develop or use case correction methods must be careful to consider negative effects on marginalized groups.
Paper Structure (24 sections, 7 equations, 11 figures, 4 tables)

This paper contains 24 sections, 7 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fairness Correction in Regression Settings
  4. Pre-processing Case Correction Methods
  5. Fairness Metrics in Regression Settings
  6. Data
  7. Case Correction Methods
  8. Method 1: Dynamics in Infection Numbers
  9. Method 2: CFR Benchmark
  10. Audit Structure: Impact of Case Correction Methods on Prediction Fairness
  11. Step one: Compute Corrected COVID-19 cases
  12. Step two: Train Quantile Regression Models
  13. Step Three: Compute Prediction Error
  14. Step Four: Assignment of County Racial/Ethnic Label
  15. Step Five: Compute Fairness Metrics
  16. ...and 9 more sections

Figures (11)

  • Figure 1: Daily cases and deaths in the United States in 2020, 7-day rolling average.
  • Figure 2: County examples of original versus corrected cases.
  • Figure 3: Method 1: a) AER values for Majority Black, Hispanic, and Non-White counties compared to majority-White counties for both uncorrected and corrected cases. b) The average and 95% confidence interval of PBL values for each majority group for lookahead 14. Baseline PBLs are all multiplied by the median county population (25,726) and Corrected PBLs are scaled by 10 to improve readability.
  • Figure 4: Method 1: a) AER values for Plurality Asian, Black, and Hispanic counties compared to plurality-White counties for both uncorrected and corrected cases. b) The average and 95% confidence interval of PBL values for each plurality group for lookahead 14. Baseline PBLs are all multiplied by the median county population (25,726) and Corrected PBLs are scaled by 10 to improve readability.
  • Figure 5: Method 2, study period 4/7/20-5/22/20: a) AER values for Majority Black, Hispanic and Non-White counties compared to majority-White counties for both uncorrected and corrected cases. b) The average and 95% confidence interval of PBL values for each majority group for lookahead 14. PBLs are all multiplied by the median county population (25,726).
  • ...and 6 more figures