Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Discretion in the Loop: Human Expertise in Algorithm-Assisted College Advising

Kara Schechtman, Benjamin Brandon, Jenise Stafford, Hannah Li, Lydia T. Liu

TL;DR

The paper examines how human discretion shapes outcomes in algorithm-assisted college advising, using a four-year randomized MAAPS trial at Georgia State University. It develops a causal graphical framework to define and audit expert targeting, and combines quantitative tests with qualitative analysis to show advisors often use non-algorithmic context to tailor interventions; about two-thirds of actions in the treatment arm appear plausibly expertly targeted. Qualitative notes reveal diverse contextual factors (personal circumstances, finances, engagement) informing decisions, and identifying advising styles that correlate with graduation. The findings underscore the importance of preserving and designing for human expertise in algorithmic decision systems, with implications for evaluation, scaling, and governance in education and other high-stakes domains.

Abstract

In higher education, many institutions use algorithmic alerts to flag at-risk students and deliver advising at scale. While much research has focused on evaluating algorithmic predictions, relatively little is known about how discretionary interventions by human experts shape outcomes in algorithm-assisted settings. We study this question using rich quantitative and qualitative data from a randomized controlled trial of an algorithm-assisted advising program at Georgia State University. Taking a mixed-methods approach, we examine whether and how advisors use context unavailable to an algorithm to guide interventions and influence student success. We develop a causal graphical framework for human expertise in the interventional setting, extending prior work on discretion in purely predictive settings. We then test a necessary condition for discretionary expertise using structured advisor logs and student outcomes data, identifying several interventions that meet the criterion for statistical significance. Accordingly, we estimate that 2 out of 3 interventions taken by advisors in the treatment arm were plausibly ``expertly targeted'' to students using non-algorithmic context. Systematic qualitative analysis of advisor notes corroborates these findings, showing a pattern of advisors incorporating diverse forms of contextual information--such as personal circumstances, financial issues, and student engagement--into their decisions. Finally, we document heterogeneity in advising styles, finding that one style elicits more holistic information about students and is associated with improved graduation rates. Our results offer theoretical and practical insight into the real-world effectiveness of algorithm-supported college advising, and underscore the importance of accounting for human expertise in the design, evaluation, and implementation of algorithmic decision systems.

Discretion in the Loop: Human Expertise in Algorithm-Assisted College Advising

TL;DR

The paper examines how human discretion shapes outcomes in algorithm-assisted college advising, using a four-year randomized MAAPS trial at Georgia State University. It develops a causal graphical framework to define and audit expert targeting, and combines quantitative tests with qualitative analysis to show advisors often use non-algorithmic context to tailor interventions; about two-thirds of actions in the treatment arm appear plausibly expertly targeted. Qualitative notes reveal diverse contextual factors (personal circumstances, finances, engagement) informing decisions, and identifying advising styles that correlate with graduation. The findings underscore the importance of preserving and designing for human expertise in algorithmic decision systems, with implications for evaluation, scaling, and governance in education and other high-stakes domains.

Abstract

In higher education, many institutions use algorithmic alerts to flag at-risk students and deliver advising at scale. While much research has focused on evaluating algorithmic predictions, relatively little is known about how discretionary interventions by human experts shape outcomes in algorithm-assisted settings. We study this question using rich quantitative and qualitative data from a randomized controlled trial of an algorithm-assisted advising program at Georgia State University. Taking a mixed-methods approach, we examine whether and how advisors use context unavailable to an algorithm to guide interventions and influence student success. We develop a causal graphical framework for human expertise in the interventional setting, extending prior work on discretion in purely predictive settings. We then test a necessary condition for discretionary expertise using structured advisor logs and student outcomes data, identifying several interventions that meet the criterion for statistical significance. Accordingly, we estimate that 2 out of 3 interventions taken by advisors in the treatment arm were plausibly ``expertly targeted'' to students using non-algorithmic context. Systematic qualitative analysis of advisor notes corroborates these findings, showing a pattern of advisors incorporating diverse forms of contextual information--such as personal circumstances, financial issues, and student engagement--into their decisions. Finally, we document heterogeneity in advising styles, finding that one style elicits more holistic information about students and is associated with improved graduation rates. Our results offer theoretical and practical insight into the real-world effectiveness of algorithm-supported college advising, and underscore the importance of accounting for human expertise in the design, evaluation, and implementation of algorithmic decision systems.
Paper Structure (95 sections, 3 theorems, 17 equations, 10 figures, 19 tables)

This paper contains 95 sections, 3 theorems, 17 equations, 10 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Contribution 1: Theoretical Model of Expertise.
  3. Contribution 2: Quantitative and Qualitative Evidence of Expertise.
  4. Related Work
  5. Causal Evaluation of Algorithm-Assisted Decision Making
  6. Predictive vs. Causal Evaluation of Algorithmic Interventions.
  7. Causal Evaluation of Learning Analytics Interventions.
  8. Human Discretion in Algorithm-Assisted Decision Making
  9. Information Asymmetries
  10. Reliance
  11. Bias, Expertise and Discretion
  12. Overview of Data and Methodology
  13. MAAPS Experiment and Data
  14. Experiment
  15. Data
  16. ...and 80 more sections

Key Result

Proposition 4.1

Suppose Assumptions assumption:dag through assumption:faithfulness are met. Expertly targeted action is nonetheless impossible to detect from facts about the observational distribution over $\mathbf{X}_t$, $A_t$ and $Y_{t+1}$ alone.

Figures (10)

  • Figure 1: Causal graphical model of a single-stage, algorithm-assisted interventional decision setting. Dotted edges represent potential functional dependencies.
  • Figure 2: Fraction of meeting comments coded with academic, non-academic, and relationship codes (Appendix \ref{['app:qual_exploratory:methods']}), broken down by advisor. Advisor 3's comments more often bring up non-academic topics and show more signs of building an advising relationship with students. Note that fractions for different code categories within each advisor may not sum to 1 as multiple codes may be applied to one meeting.
  • Figure 3: Histograms of applied interventions per advisor, demonstrating an especially low diversity of interventions applied by Advisor 1. Advisors 4 and 5 are excluded due to the limited number of available comments for each.
  • Figure 5: Enrolled students per semester
  • Figure 6: Number of meetings per semester
  • ...and 5 more figures

Theorems & Definitions (10)

  • Definition 4.1: Expertly targeted action
  • Proposition 4.1
  • proof : Proof Sketch
  • Proposition 4.2
  • proof
  • Definition 4.2
  • Definition 4.3
  • Proposition 4.3
  • proof
  • proof