Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing AI Assisted Writing with One-Shot Implicit Negative Feedback

Benjamin Towle, Ke Zhou

TL;DR

Nifty is introduced, an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process and finds up to 34% improvement in Rouge-L, 89% improvement in generating the correct intent, and an 86% win-rate according to human evaluators compared to a vanilla AI writing system on the MultiWOZ and Schema-Guided Dialog datasets.

Abstract

AI-mediated communication enables users to communicate more quickly and efficiently. Various systems have been proposed such as smart reply and AI-assisted writing. Yet, the heterogeneity of the forms of inputs and architectures often renders it challenging to combine insights from user behaviour in one system to improve performance in another. In this work, we consider the case where the user does not select any of the suggested replies from a smart reply system, and how this can be used as one-shot implicit negative feedback to enhance the accuracy of an AI writing model. We introduce Nifty, an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process. Empirically, we find up to 34% improvement in Rouge-L, 89% improvement in generating the correct intent, and an 86% win-rate according to human evaluators compared to a vanilla AI writing system on the MultiWOZ and Schema-Guided Dialog datasets.

Enhancing AI Assisted Writing with One-Shot Implicit Negative Feedback

TL;DR

Nifty is introduced, an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process and finds up to 34% improvement in Rouge-L, 89% improvement in generating the correct intent, and an 86% win-rate according to human evaluators compared to a vanilla AI writing system on the MultiWOZ and Schema-Guided Dialog datasets.

Abstract

AI-mediated communication enables users to communicate more quickly and efficiently. Various systems have been proposed such as smart reply and AI-assisted writing. Yet, the heterogeneity of the forms of inputs and architectures often renders it challenging to combine insights from user behaviour in one system to improve performance in another. In this work, we consider the case where the user does not select any of the suggested replies from a smart reply system, and how this can be used as one-shot implicit negative feedback to enhance the accuracy of an AI writing model. We introduce Nifty, an approach that uses classifier guidance to controllably integrate implicit user feedback into the text generation process. Empirically, we find up to 34% improvement in Rouge-L, 89% improvement in generating the correct intent, and an 86% win-rate according to human evaluators compared to a vanilla AI writing system on the MultiWOZ and Schema-Guided Dialog datasets.

Paper Structure

This paper contains 21 sections, 3 equations, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Filtering with User Simulator
  5. Smart Reply
  6. Generator
  7. Classifier Guidance
  8. Experiment
  9. Datasets
  10. Metrics
  11. Baselines
  12. Model Details
  13. Main Results
  14. Varying Degree of Guidance
  15. Human Evaluation
  16. ...and 6 more sections

Figures (1)

  • Figure 1: Example of how an agent may utilise either a smart reply system or an AI writing system to speed up communication with a customer. Our approach, Nifty, uses implicit negative feedback from the rejected suggestions to improve the AI writer's prediction.