Challenging the Human-in-the-loop in Algorithmic Decision-making

TL;DR

This paper challenges the assumption that human oversight alone ensures ethical and effective algorithmic decision-making by highlighting two distinct human roles—the Strategic Decision-Maker (SDM) and the Practical Decision-Maker (PDM)—and how misalignment between their goals can shift power within ADM. It develops a conceptual framework and empirically demonstrates, on a machine-learning benchmark, that a PDM can meaningfully alter outcomes even with limited deviations from algorithmic recommendations, potentially undermining the SDM’s intended societal values. The work emphasizes the need for explicit information exchange, tailored explanations for each stakeholder, and robust planning to account for the PDM’s additional influence, including cooperative and multi-PDM scenarios. It also discusses ethical, governance, and transparency implications for public-service ADM, suggesting design principles and explanation strategies that acknowledge the dual-hil dynamics and aim to preserve the SDM’s value-driven objectives. The results underscore the practical importance of value-aligned explanations and coordination mechanisms to prevent unintended value drift in socio-technical ADM systems.

Abstract

We discuss the role of humans in algorithmic decision-making (ADM) for socially relevant problems from a technical and philosophical perspective. In particular, we illustrate tensions arising from diverse expectations, values, and constraints by and on the humans involved. To this end, we assume that a strategic decision-maker (SDM) introduces ADM to optimize strategic and societal goals while the algorithms' recommended actions are overseen by a practical decision-maker (PDM) - a specific human-in-the-loop - who makes the final decisions. While the PDM is typically assumed to be a corrective, it can counteract the realization of the SDM's desired goals and societal values not least because of a misalignment of these values and unmet information needs of the PDM. This has significant implications for the distribution of power between the stakeholders in ADM, their constraints, and information needs. In particular, we emphasize the overseeing PDM's role as a potential political and ethical decision maker, who acts expected to balance strategic, value-driven objectives and on-the-ground individual decisions and constraints. We demonstrate empirically, on a machine learning benchmark dataset, the significant impact an overseeing PDM's decisions can have even if the PDM is constrained to performing only a limited amount of actions differing from the algorithms' recommendations. To ensure that the SDM's intended values are realized, the PDM needs to be provided with appropriate information conveyed through tailored explanations and its role must be characterized clearly. Our findings emphasize the need for an in-depth discussion of the role and power of the PDM and challenge the often-taken view that just including a human-in-the-loop in ADM ensures the 'correct' and 'ethical' functioning of the system.

Figures (4)

• Figure 1: Considered adm process. The sdm makes long-term decisions regarding the process, e.g., which societal values it should realize, while the pdm oversees decisions on individuals. The adm makes recommendations for treatments of individuals based on an individual's score predicted from an individual's features. The actual applied treatment is selected by the pdm and results in an outcome for the individual. To realize the sdm's intended values, the sdm and the pdm need to account for each other and act accordingly while both might base their decisions on different information available to them.
• Figure 2: adm framework with sdm () and pdm (). Explanations for different parts of the framework are marked by colored boxes. The circled nodes represent the true state of an individual $S_t^i$, its observed features $X_t^i$ and confounding factors $H_t^i$, the prediction by an algorithm $\mathcal{A}$ for the individual $P_t^i$, the suggested treatment $\tilde{T}_t^i$ according to the algorithm, and the next state of the individual $S_{t+1}^i$. This next state also depends on hidden confounders $H_t^\textnormal{SC}$ independent of an individual, e.g., changes in laws. The pdm can alter the treatment, becoming $T_t^i$. Algorithm $\mathcal{A}$ is selected to realize societal and ethical values.
• Figure 3: Different possible weightings for societal values can lead to the same decisions if the pdm can make decisions that deviate from the algorithm's recommendation. Sensitive attribute 1 is "proportion of blacks by town" and sensitive attribute 2 is "full-value property-tax rate per $10,000". (\ref{['fig:possible-values-scatter']}) Different values realizing the same decisions in up to $1\%$ of the cases. We can understand this as the pdm blurring the sdm's intent, or the sdm giving up part of its power in favor of control of the system through the pdm. (\ref{['fig:possible-values-heatmap']}) Sum of different decisions for deviations of the sdm's and pdm's values.
• Figure 4: Change of realized societal values when correcting the recommended actions using the ground truth. A degradation of values can be observed for an increasing number of corrections which can be attributed to the bias in the dataset. Sensitive attribute 1 is "proportion of blacks by town" and sensitive attribute 2 is "full-value property-tax rate per $10,000".

Theorems & Definitions (1)

• proof