Designing Shared Information Displays for Agents of Varying Strategic Sophistication

TL;DR

This work tackles distribution shift in data-driven predictions within strategic environments by isolating how interface design choices—specifically uncertainty visualization and post hoc prediction error feedback—shape decisions across agents with varying level-$k$ sophistication. It employs a large, online, staged experiment in a three-action congestion game, using a Poisson-CH model to endow participants with L0–L2 thinking and a counterfactual taxi-flow model to generate realistic payoffs. Key findings reveal trade-offs: interfaces that improve individual decision quality may hurt social welfare, while reductions in distribution shift can bolster perceived reliability and trust in the display, particularly for more sophisticated agents. The results advance understanding of how to design prediction interfaces for strategic settings and open avenues for balancing individual utility with collective welfare in shared-information environments.

Abstract

Data-driven predictions are often perceived as inaccurate in hindsight due to behavioral responses. In this study, we explore the role of interface design choices in shaping individuals' decision-making processes in response to predictions presented on a shared information display in a strategic setting. We introduce a novel staged experimental design to investigate the effects of design features, such as visualizations of prediction uncertainty and error, within a repeated congestion game. In this game, participants assume the role of taxi drivers and use a shared information display to decide where to search for their next ride. Our experimental design endows agents with varying level-$k$ depths of thinking, allowing some agents to possess greater sophistication in anticipating the decisions of others using the same information display. Through several extensive experiments, we identify trade-offs between displays that optimize individual decisions and those that best serve the collective social welfare of the system. We find that the influence of display characteristics varies based on an agent's strategic sophistication. We observe that design choices promoting individual-level decision-making can lead to suboptimal system outcomes, as manifested by a lower realization of potential social welfare. However, this decline in social welfare is offset by a reduction in the distribution shift, narrowing the gap between predicted and realized system outcomes, which potentially enhances the perceived reliability and trustworthiness of the information display post hoc. Our findings pave the way for new research questions concerning the design of effective prediction interfaces in strategic environments.

Figures (11)

• Figure 1: Diagram of key features of our experimental design. (1) We define a pseudo-Poisson distribution using a $\text{Poisson}(1.5)$, which includes L0s-L2s. (2) We endow level-specific beliefs by normalizing the pseudo-Poisson distribution for L1s and L2s who are our study participants. (3) We conduct a staged data collection: in each trial, participants use the information display (i.e., shown in \ref{['fig:prediction-display']}) to make decisions and then review level-specific feedback (i.e., shown in \ref{['fig:feedback-page']}). L1s and L2s complete all trials in the same order, but L1s complete the study before L2s so L1s' responses can be used with that of L0s (i.e., the taxi data) to construct level-specific feedback that aligns with L2s' endowed belief. (4) We calculate the system outcome by combining decisions from L0s (i.e., the taxi data) and L1-L2s (i.e., the collected responses) according to the mixture we used to define the level distribution (i.e., step 1). (5) We conduct two replications of the experiment by varying the trial order and redefining the level mixture of L0s-L2s using a $\text{Poisson}(3)$.
• Figure 2: The interface used to collect participants' decisions. When providing anticipation, after a participant provides guesses for two districts, the interface imputes the flow of the last district to ensure proper summation of the total number of drivers of the decision scenario. The interface dynamically updates the current flow sum and the amount of flow to be allocated or removed if the elicited flows do not sum to the correct total.
• Figure 3: Examples of the information displays used for decision scenarios varied by uncertainty quantification. Right: NetHOPs, which render 1,000 hypothetical outcomes, are presented in a looping animation with an animation speed of 0.2 seconds per frame using a fixed force-directed layout (i.e., anchoring $\alpha = 1$). This approach follows suggestions from zhang2021visualizing that best support node-attribute and link-attribute tasks. Left: Point estimates where the rendered payoffs are the weighted averages of all simulations.
• Figure 4: An example of bandit and full feedback provided after participants submit their responses. Both feedback types display the decision result and current study compensation (i.e., highlighted in red). Bandit feedback exclusively provides this information. Full feedback additionally presents three visualizations, reminding participants of the prediction they used (left), the level-specific outcome (middle), and their anticipation submitted with the decision (right).
• Figure 5: (A) Median point estimates of the expected welfare ratio (top) and distribution shift (bottom) resulted from the system outcome. We expressed uncertainty as 95% credible intervals (CIs) predicted by the fixed effects of both aggregate-level models for each treatment marginalized over trials. (B) Same as (A), we present both the point estimate and the corresponding 95% CIs for the welfare ratio (top) and distribution shift (bottom) for each trial to examine changes in system outcome by trial.