People Can Accurately Predict Behavior of Complex Algorithms That Are Available, Compact, and Aligned

TL;DR

The paper introduces ACA (Availability, Compactness, Alignment) as a triad determining when users can form accurate predictive mental models of complex algorithms. Through a preregistered, large-scale study ($N=1250$) in social-media feed ranking with $25$ algorithm conditions, it shows that only ACA-compliant algorithms yield high prediction accuracy (about $85 ext{\%}$) while violations degrade to near baseline, with substantial effects driven by a three-way interaction ($A\cdot C\cdot A$) having coefficient $1.826$ and odds ratio $6.2$. The authors link mental-model accuracy to alignment between user conjectures and actual algorithmic execution, and reveal that users often rely on available and compact concepts, sometimes using non-ideal or fragmented theories under uncertainty. The findings have practical implications for designing interpretable, user-predictable, high-performance algorithms in social media and beyond, including guidance for explainability, user education, and participatory design.

Abstract

Users trust algorithms more when they can predict the algorithms' behavior. Simple algorithms trivially yield predictively accurate mental models, but modern AI algorithms have often been assumed too complex for people to build predictive mental models, especially in the social media domain. In this paper, we describe conditions under which even complex algorithms can yield predictive mental models, opening up opportunities for a broader set of human-centered algorithms. We theorize that users will form an accurate predictive mental model of an algorithm's behavior if and only if the algorithm simultaneously satisfies three criteria: (1) cognitive availability of the underlying concepts being modeled, (2) concept compactness (does it form a single cognitive construct?), and (3) high alignment between the person's and algorithm's execution of the concept. We evaluate this theory through a pre-registered experiment (N=1250) where users predict behavior of 25 social media feed ranking algorithms that vary on these criteria. We find that even complex (e.g., LLM-based) algorithms enjoy accurate prediction rates when they meet all criteria, and even simple (e.g., basic term count) algorithms fail to be predictable when a single criterion fails. We also find that these criteria determine outcomes beyond prediction accuracy, such as which mental models users deploy to make their predictions.

Figures (8)

• Figure 1: ACA theory: Algorithms satisfying the criteria of Availability, Compactness, and Alignment have the capacity to be predictable, even if the algorithm internal operation is opaque (e.g., in the case of complex deep learning models). These three criteria are mutually necessary in order for people to form accurate predictive mental models
• Figure 2: The process we use for measuring prediction accuracy takes place in three of the five study phases: exposure, training, and testing. In exposure, participants look at a particular feed and try to figure what the algorithm ranking it does. In training, they are given feedback on whether they correctly rank the pairs of posts they are given. In testing, they repeat the ranking task, but without feedback.
• Figure 3: Participants predicted algorithm behavior with the highest accuracy for algorithm conditions that satisfied all three ACA criteria. Most algorithms that failed one or more criteria had predictive performance at close to baseline rates.
• Figure 4: Participants predicted algorithm behavior at close to 80% or above for algorithms that satisfied all three ACA criteria. Most algorithms that failed one or more criteria had predictive performance within 10% of a random guessing baseline (50%) rate. Only two non-ACA conditions (noisy likes and time-related) had prediction accuracy at slightly above 60%.
• Figure 5: We visualize the percentage of mental models for each algorithm condition that matched the design of the algorithm. The ACA conditions exhibited higher levels of matching, though the writing quality condition had an unusually low level of complete matching compared to the other ACA conditions.