Dynamic Information Sub-Selection for Decision Support

TL;DR

Empirical validation of the proposed DISS methodology shows superior performance to state-of-the-art methods across various applications, including biased decision-maker support, expert assignment optimization, large language model decision support, and interpretability.

Abstract

We introduce Dynamic Information Sub-Selection (DISS), a novel framework of AI assistance designed to enhance the performance of black-box decision-makers by tailoring their information processing on a per-instance basis. Blackbox decision-makers (e.g., humans or real-time systems) often face challenges in processing all possible information at hand (e.g., due to cognitive biases or resource constraints), which can degrade decision efficacy. DISS addresses these challenges through policies that dynamically select the most effective features and options to forward to the black-box decision-maker for prediction. We develop a scalable frequentist data acquisition strategy and a decision-maker mimicking technique for enhanced budget efficiency. We explore several impactful applications of DISS, including biased decision-maker support, expert assignment optimization, large language model decision support, and interpretability. Empirical validation of our proposed DISS methodology shows superior performance to state-of-the-art methods across various applications.

Figures (16)

• Figure 1: Given an instance, the DISS policy selects a subset of features (and options) to forward to a decision-maker to render its decisions. A reward that evaluates the quality of the decision maker's decision is used as feedback on the DISS policy.
• Figure 1: Average rewards vs. observation data budget in the overload bias experimental setting, including additional baseline results for CMTS in 'skin' (using a Gaussian Process, not able to achieve rewards in range).
• Figure 2: Average rewards vs. observation data budget on cognitive-overloaded expert environment.
• Figure 2: Average rewards vs. observation data budget in the risk aversion bias experimental setting, including additional baseline results for CMTS in 'skin' (using a Gaussian Process).
• Figure 3: Average rewards vs. observation data budget on risk-averse expert environment (overly concerned with minimizing false negative predictions).