Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

Shuai Ma, Qiaoyi Chen, Xinru Wang, Chengbo Zheng, Zhenhui Peng, Ming Yin, Xiaojuan Ma

TL;DR

This work introduces Human-AI Deliberation, a framework that enables structured deliberation between humans and AI to resolve conflicting opinions in decision-making. It pairs a Deliberative AI, built from a DS-model and an LLM bridge, with a WoE-based architecture to elicit thoughts, align perspectives, discuss, and update conclusions. An illustrative graduate admissions task demonstrates that Deliberative AI can improve decision accuracy and promote more appropriate human reliance than conventional XAI, while also highlighting trade-offs in user experience and trust. The study provides empirical and qualitative insights into design considerations, potential benefits, and challenges for integrating deliberation into AI-assisted decision tools, with implications for future high-stakes domains. Overall, the paper advances a practical blueprint for building deliberation-enabled AI systems that support more thoughtful, transparent, and collaborative decision-making workflows.

Abstract

In AI-assisted decision-making, humans often passively review AI's suggestion and decide whether to accept or reject it as a whole. In such a paradigm, humans are found to rarely trigger analytical thinking and face difficulties in communicating the nuances of conflicting opinions to the AI when disagreements occur. To tackle this challenge, we propose Human-AI Deliberation, a novel framework to promote human reflection and discussion on conflicting human-AI opinions in decision-making. Based on theories in human deliberation, this framework engages humans and AI in dimension-level opinion elicitation, deliberative discussion, and decision updates. To empower AI with deliberative capabilities, we designed Deliberative AI, which leverages large language models (LLMs) as a bridge between humans and domain-specific models to enable flexible conversational interactions and faithful information provision. An exploratory evaluation on a graduate admissions task shows that Deliberative AI outperforms conventional explainable AI (XAI) assistants in improving humans' appropriate reliance and task performance. Based on a mixed-methods analysis of participant behavior, perception, user experience, and open-ended feedback, we draw implications for future AI-assisted decision tool design.

Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making

TL;DR

This work introduces Human-AI Deliberation, a framework that enables structured deliberation between humans and AI to resolve conflicting opinions in decision-making. It pairs a Deliberative AI, built from a DS-model and an LLM bridge, with a WoE-based architecture to elicit thoughts, align perspectives, discuss, and update conclusions. An illustrative graduate admissions task demonstrates that Deliberative AI can improve decision accuracy and promote more appropriate human reliance than conventional XAI, while also highlighting trade-offs in user experience and trust. The study provides empirical and qualitative insights into design considerations, potential benefits, and challenges for integrating deliberation into AI-assisted decision tools, with implications for future high-stakes domains. Overall, the paper advances a practical blueprint for building deliberation-enabled AI systems that support more thoughtful, transparent, and collaborative decision-making workflows.

Abstract

In AI-assisted decision-making, humans often passively review AI's suggestion and decide whether to accept or reject it as a whole. In such a paradigm, humans are found to rarely trigger analytical thinking and face difficulties in communicating the nuances of conflicting opinions to the AI when disagreements occur. To tackle this challenge, we propose Human-AI Deliberation, a novel framework to promote human reflection and discussion on conflicting human-AI opinions in decision-making. Based on theories in human deliberation, this framework engages humans and AI in dimension-level opinion elicitation, deliberative discussion, and decision updates. To empower AI with deliberative capabilities, we designed Deliberative AI, which leverages large language models (LLMs) as a bridge between humans and domain-specific models to enable flexible conversational interactions and faithful information provision. An exploratory evaluation on a graduate admissions task shows that Deliberative AI outperforms conventional explainable AI (XAI) assistants in improving humans' appropriate reliance and task performance. Based on a mixed-methods analysis of participant behavior, perception, user experience, and open-ended feedback, we draw implications for future AI-assisted decision tool design.
Paper Structure (48 sections, 1 equation, 11 figures, 2 tables)

This paper contains 48 sections, 1 equation, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI-Assisted Decision-Making: Objectives and Challenges
  4. The Role of Deliberation in Human Decision Making
  5. Existing Studies on Deliberation in AI-Assisted Decision Making
  6. Deliberative AI for Human-AI Deliberation
  7. Integrating Deliberation into Decision-Making
  8. An Architecture of Human-AI Deliberation
  9. Deliberative AI: Design Considerations and Overall Structure
  10. Instantiating the Architecture: Graduate Admission Prediction
  11. Task, Dataset and AI Model
  12. Human-AI Thought Representation and Alignment
  13. Implementation of Deliberative AI
  14. I. Communication Layer.
  15. II. Control Layer.
  16. ...and 33 more sections

Figures (11)

  • Figure 1: An illustration of Human-AI Deliberation. (A) In traditional AI-assisted decision-making, when humans disagree with AI's suggestions (and only find parts of AI's reasons applaudable), it is difficult for humans to decide whether and how much to adopt AI's suggestion. (B) In our proposed Human-AI Deliberation, we provide opportunities for the human and the AI model to deliberate on conflicting opinions by discussing related evidence and arguments. Then, AI and humans can update their thoughts (when find it necessary) and reach final predictions.
  • Figure 2: The architecture of Human-AI Deliberation. (A) Illustrates the Weight of Evidence (WoE) concept in decision-making, showcasing how decision-makers assess evidence across dimensions to shape opinions and arrive at a final decision. (B) Presents the Architecture for Human-AI Deliberation, with key activities (shown in grey boxes) and potential design space (shown in dashed-line boxes).
  • Figure 3: The architecture of Deliberative AI which integrates a domain-specific model and a Large Language Model, enabling the AI to engage in natural communication with humans while also harnessing domain knowledge derived from the specialized model.
  • Figure 4: The conversation flow for the deliberative discussion.
  • Figure 5: The interface of Deliberative AI. The interface contains three parts. The top part (A) is the applicant's profile. The bottom left part (B) is the region for humans and AI to indicate (and update) their opinions. The bottom right part (C) is the discussion region where humans and AI can discuss conflicting opinions. (All the dashed lines are only for illustration)
  • ...and 6 more figures