Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents

Nikhil Sharma, Zheng Zhang, Daniel Lee, Namita Krishnan, Guang-Jie Ren, Ziang Xiao, Yunyao Li

TL;DR

The paper addresses the problem that user feedback to Conversational Agents is often sparse and low-quality, hindering alignment and learning. It combines a formative study grounded in Gricean maxims to identify four Feedback Barriers (Common Ground, Verifiability, Communication, Informativeness) and develops FeedbackGPT, a scaffolded, model-agnostic interface designed to mitigate these barriers via persistent shared frames, proactive guidance, and verifiable reasoning. A second within-subject study shows that these scaffolds increase goal-referenced, actionable, and progressively rich feedback, albeit with higher cognitive load. The work provides design principles and concrete interface scaffolds to foster truly collaborative human–AI interaction and calls for broader model-level advances (memory, calibration, multi-turn training) to realize robust, reciprocal feedback in practice.

Abstract

High-quality feedback is essential for effective human-AI interaction. It bridges knowledge gaps, corrects digressions, and shapes system behavior; both during interaction and throughout model development. Yet despite its importance, human feedback to AI is often infrequent and low quality. This gap motivates a critical examination of human feedback during interactions with AIs. To understand and overcome the challenges preventing users from giving high-quality feedback, we conducted two studies examining feedback dynamics between humans and conversational agents (CAs). Our formative study, through the lens of Grice's maxims, identified four Feedback Barriers -- Common Ground, Verifiability, Communication, and Informativeness -- that prevent high-quality feedback by users. Building on these findings, we derive three design desiderata and show that systems incorporating scaffolds aligned with these desiderata enabled users to provide higher-quality feedback. Finally, we detail a call for action to the broader AI community for advances in Large Language Models capabilities to overcome Feedback Barriers.

Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents

TL;DR

The paper addresses the problem that user feedback to Conversational Agents is often sparse and low-quality, hindering alignment and learning. It combines a formative study grounded in Gricean maxims to identify four Feedback Barriers (Common Ground, Verifiability, Communication, Informativeness) and develops FeedbackGPT, a scaffolded, model-agnostic interface designed to mitigate these barriers via persistent shared frames, proactive guidance, and verifiable reasoning. A second within-subject study shows that these scaffolds increase goal-referenced, actionable, and progressively rich feedback, albeit with higher cognitive load. The work provides design principles and concrete interface scaffolds to foster truly collaborative human–AI interaction and calls for broader model-level advances (memory, calibration, multi-turn training) to realize robust, reciprocal feedback in practice.

Abstract

High-quality feedback is essential for effective human-AI interaction. It bridges knowledge gaps, corrects digressions, and shapes system behavior; both during interaction and throughout model development. Yet despite its importance, human feedback to AI is often infrequent and low quality. This gap motivates a critical examination of human feedback during interactions with AIs. To understand and overcome the challenges preventing users from giving high-quality feedback, we conducted two studies examining feedback dynamics between humans and conversational agents (CAs). Our formative study, through the lens of Grice's maxims, identified four Feedback Barriers -- Common Ground, Verifiability, Communication, and Informativeness -- that prevent high-quality feedback by users. Building on these findings, we derive three design desiderata and show that systems incorporating scaffolds aligned with these desiderata enabled users to provide higher-quality feedback. Finally, we detail a call for action to the broader AI community for advances in Large Language Models capabilities to overcome Feedback Barriers.
Paper Structure (79 sections, 1 equation, 3 figures, 1 table)

This paper contains 79 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Feedback in Human–AI Interaction
  4. User interaction in LLM-based systems
  5. Human–AI Interaction with LLM-Powered Conversational Agents
  6. The Science of Effective Feedback
  7. Formative Study
  8. Method and Participants
  9. Analysis
  10. Positionality Statement
  11. Findings - Descriptive
  12. What are the common breakdowns while interacting with CAs?
  13. How do users give feedback?
  14. Does user feedback follow the properties of high-quality feedback?
  15. Findings - What prevents users from giving "high-quality" feedback?
  16. ...and 64 more sections

Figures (3)

  • Figure 1: Overview of FeedbackGPT. The system consists of three new interface components, a sidebar, a panel and a popup. The interface give users seven different scaffolds for the three desideratum as described in Sec \ref{['sec:operationalizing']}
  • Figure 2: Overall study procedure for Study 2. In the pre-task survey, participants answered questions regarding their prior experience with conversational AI and their prior attitude and demographic questions. Participants then chose a task from three available topics. They were then randomly assigned a system with which they complete the task. Following which, they were asked three post-task questions about their results, conversation and productivity. They then choose a second task from the remaining two options and were assigned the system not assigned in task 1 and completed the task with the post task questions. After which, participants were asked three general questions about their experience with FeedbackGPT and then were asked eight questions about their experience with FeedbackGPT aimed to capture the effects of feedback barriers in FeedbackGPT.
  • Figure 3: Scatterplot showing the distribution of user feedbacks. The x axis are the ChatGPT rate of giving high quality feedback and the Y axis are FeedbackGPT rate of giving high quality feedback per 100 feedbacks. We see FeedbackGPT made feedback more goal-referenced, actionable and progressive. However, users still struggle with producing articulate feedback.