Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conversational AI: The Science Behind the Alexa Prize

Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, Art Pettigrue

TL;DR

The advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI are outlined.

Abstract

Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions. To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the efforts of participating teams, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability. This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.

Conversational AI: The Science Behind the Alexa Prize

TL;DR

The advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI are outlined.

Abstract

Conversational agents are exploding in popularity. However, much work remains in the area of social conversation as well as free-form conversation over a broad range of domains and topics. To advance the state of the art in conversational AI, Amazon launched the Alexa Prize, a 2.5-million-dollar university competition where sixteen selected university teams were challenged to build conversational agents, known as socialbots, to converse coherently and engagingly with humans on popular topics such as Sports, Politics, Entertainment, Fashion and Technology for 20 minutes. The Alexa Prize offers the academic community a unique opportunity to perform research with a live system used by millions of users. The competition provided university teams with real user conversational data at scale, along with the user-provided ratings and feedback augmented with annotations by the Alexa team. This enabled teams to effectively iterate and make improvements throughout the competition while being evaluated in real-time through live user interactions. To build their socialbots, university teams combined state-of-the-art techniques with novel strategies in the areas of Natural Language Understanding, Context Modeling, Dialog Management, Response Generation, and Knowledge Acquisition. To support the efforts of participating teams, the Alexa Prize team made significant scientific and engineering investments to build and improve Conversational Speech Recognition, Topic Tracking, Dialog Evaluation, Voice User Experience, and tools for traffic management and scalability. This paper outlines the advances created by the university teams as well as the Alexa Prize team to achieve the common goal of solving the problem of Conversational AI.

Paper Structure

This paper contains 29 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Customer Experience
  3. CX for Alexa Users
  4. CX for University Teams
  5. Alexa Prize Summit
  6. Shared Resources and Tools
  7. Socialbot Management Framework
  8. Invocation and Feedback Framework
  9. Uptime Measurement and Availability Monitoring
  10. Managing Customer Experience (CX)
  11. Architecture
  12. Summary of architectural approaches
  13. Scalability challenges
  14. Science
  15. Conversational Automatic Speech Recognition
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Conversational ASR performance improvement: Relative reduction in WER with respect to base model.
  • Figure 2: Daily Ratings for socialbots during the competition.
  • Figure 3: Conversation Duration Median and 90th Percentile for socialbots during the competition.
  • Figure 4: Conversation Duration 90th Percentile for socialbots during the competition.
  • Figure 5: Response Error Rate (RER) for individual utterance-response pairs for All Socialbots and Finalists during the competition.