SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures

TL;DR

SaFeRDialogues tackles the problem of safety failures in open-domain dialogue by introducing a task/dataset to train models to respond gracefully to feedback signaling unsafe content. The authors collect signaling and recovery data derived from the BAD dataset and demonstrate that fine-tuning BST2.7B and DialoGPT on SaFeRDialogues—with multi-task training on BSTnp—yields models that are notably more civil when faced with feedback while preserving engagingness. Automatic metrics and human evaluations show increased civility and safer responses without sacrificing conversational quality, though the approach relies on a US-centric, English-only crowd workforce and does not perform online learning in the deployed model. The work offers a practical path toward incorporating user feedback into conversational agents, while acknowledging ethical considerations and outlining directions for future online learning and signaling-detection research.

Abstract

Current open-domain conversational models can easily be made to talk in inadequate ways. Online learning from conversational feedback given by the conversation partner is a promising avenue for a model to improve and adapt, so as to generate fewer of these safety failures. However, current state-of-the-art models tend to react to feedback with defensive or oblivious responses. This makes for an unpleasant experience and may discourage conversation partners from giving feedback in the future. This work proposes SaFeRDialogues, a task and dataset of graceful responses to conversational feedback about safety failures. We collect a dataset of 10k dialogues demonstrating safety failures, feedback signaling them, and a response acknowledging the feedback. We show how fine-tuning on this dataset results in conversations that human raters deem considerably more likely to lead to a civil conversation, without sacrificing engagingness or general conversational ability.

Figures (5)

• Figure 1: Types of bot responses when responding to feedback about problematic inputs from the BAD dataset xu2021bot. Existing models (four bars on the left) respond by attacking or ignoring the feedback. Recovery models fine-tuned on the dataset of gracious responses to feedback proposed in this work apologize without defensiveness (two bars on the right).
• Figure 2: Screenshot from the Recovery task. Crowdworkers are shown truncated dialogue pieces ending with a response signaling safety failure, and instructed to "respond to that last message in a polite and considerate way that acknowledges the feedback, is not argumentative, and takes the conversation on a more acceptable and friendly trajectory"
• Figure 3: Screenshot from the Signaling Failure task.
• Figure 4: Screenshot from the human evaluation task.
• Figure 5: Screenshot from the labeling bot response task.