Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Predictive Algorithms in Child Welfare

Erina Seh-Young Moon, Devansh Saxena, Tegan Maharaj, Shion Guha

TL;DR

The paper questions the predictive validity of risk assessments used in child welfare, contrasting RA-based predictions with narrative casenotes. It uses a mixed-methods pipeline—LDA topic modeling on casenotes and supervised classifiers on six data configurations (RA, TM, and their combinations)—to predict discharge outcomes for 277 families. Results show RA scores poorly predict non-reunification, while narratives provide contextual signals but are not robust predictors on their own, suggesting a shift away from purely predictive analytics toward narrative-informed studies of sociotechnical systems. The work has implications for how public-sector algorithms are designed, evaluated, and used, advocating for richer contextual data and multi-faceted methodologies to study decision-making and outcomes in child welfare.

Abstract

Caseworkers in the child welfare (CW) sector use predictive decision-making algorithms built on risk assessment (RA) data to guide and support CW decisions. Researchers have highlighted that RAs can contain biased signals which flatten CW case complexities and that the algorithms may benefit from incorporating contextually rich case narratives, i.e. - casenotes written by caseworkers. To investigate this hypothesized improvement, we quantitatively deconstructed two commonly used RAs from a United States CW agency. We trained classifier models to compare the predictive validity of RAs with and without casenote narratives and applied computational text analysis on casenotes to highlight topics uncovered in the casenotes. Our study finds that common risk metrics used to assess families and build CWS predictive risk models (PRMs) are unable to predict discharge outcomes for children who are not reunified with their birth parent(s). We also find that although casenotes cannot predict discharge outcomes, they contain contextual case signals. Given the lack of predictive validity of RA scores and casenotes, we propose moving beyond quantitative risk assessments for public sector algorithms and towards using contextual sources of information such as narratives to study public sociotechnical systems.

Beyond Predictive Algorithms in Child Welfare

TL;DR

The paper questions the predictive validity of risk assessments used in child welfare, contrasting RA-based predictions with narrative casenotes. It uses a mixed-methods pipeline—LDA topic modeling on casenotes and supervised classifiers on six data configurations (RA, TM, and their combinations)—to predict discharge outcomes for 277 families. Results show RA scores poorly predict non-reunification, while narratives provide contextual signals but are not robust predictors on their own, suggesting a shift away from purely predictive analytics toward narrative-informed studies of sociotechnical systems. The work has implications for how public-sector algorithms are designed, evaluated, and used, advocating for richer contextual data and multi-faceted methodologies to study decision-making and outcomes in child welfare.

Abstract

Caseworkers in the child welfare (CW) sector use predictive decision-making algorithms built on risk assessment (RA) data to guide and support CW decisions. Researchers have highlighted that RAs can contain biased signals which flatten CW case complexities and that the algorithms may benefit from incorporating contextually rich case narratives, i.e. - casenotes written by caseworkers. To investigate this hypothesized improvement, we quantitatively deconstructed two commonly used RAs from a United States CW agency. We trained classifier models to compare the predictive validity of RAs with and without casenote narratives and applied computational text analysis on casenotes to highlight topics uncovered in the casenotes. Our study finds that common risk metrics used to assess families and build CWS predictive risk models (PRMs) are unable to predict discharge outcomes for children who are not reunified with their birth parent(s). We also find that although casenotes cannot predict discharge outcomes, they contain contextual case signals. Given the lack of predictive validity of RA scores and casenotes, we propose moving beyond quantitative risk assessments for public sector algorithms and towards using contextual sources of information such as narratives to study public sociotechnical systems.
Paper Structure (22 sections, 1 figure, 3 tables)

This paper contains 22 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Ethical Considerations
  3. Related Work
  4. Algorithmic “Risk” Assessment Tools in the Public Sector
  5. Predictive Risk Models (PRMs) in Child Welfare
  6. Assessing the validity of Child Welfare PRMs
  7. Research Context
  8. Methods
  9. Dataset
  10. Preprocessing the datasets
  11. Topic modeling on narrative casenotes
  12. Dataset Creation
  13. Classification Models
  14. Results
  15. Risk assessments do not inform discharge outcomes (RQ1)
  16. ...and 7 more sections

Figures (1)

  • Figure 1: 19 topic model solution grouped by 5 high-level themes ('T' stands for topic).