Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

Rafael Ferreira, Diogo Tavares, Diogo Silva, Rodrigo Valério, João Bordalo, Inês Simões, Vasco Ramos, David Semedo, João Magalhães

TL;DR

TWIZ v2 advances multimodal conversational AI for real-world manual tasks by integrating a modular, task-grounded architecture with a specialized TWIZ-LLM. It advances humanly-shaped conversations through frictionless task discovery, creative customization (Creative Cooking), and a promoter mechanism to entice task initiation, while enabling zero-shot dialogue flows via DST framing and zero-shot prompts. The multimodal emphasis spans generative visual task/step illustrations, video moment retrieval, and curiosity generation grounded by external sources, all reinforced by guardrails and verification (NL2VI, VQA, Truetrue). Across extensive automatic and human evaluations, TWIZ demonstrates robust task guidance, engaging stimuli, and usable UX, informing scalable, trustworthy TaskBots with practical impact for end-users and developers alike.

Abstract

In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks. Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.

TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

TL;DR

TWIZ v2 advances multimodal conversational AI for real-world manual tasks by integrating a modular, task-grounded architecture with a specialized TWIZ-LLM. It advances humanly-shaped conversations through frictionless task discovery, creative customization (Creative Cooking), and a promoter mechanism to entice task initiation, while enabling zero-shot dialogue flows via DST framing and zero-shot prompts. The multimodal emphasis spans generative visual task/step illustrations, video moment retrieval, and curiosity generation grounded by external sources, all reinforced by guardrails and verification (NL2VI, VQA, Truetrue). Across extensive automatic and human evaluations, TWIZ demonstrates robust task guidance, engaging stimuli, and usable UX, informing scalable, trustworthy TaskBots with practical impact for end-users and developers alike.

Abstract

In this report, we describe the vision, challenges, and scientific contributions of the Task Wizard team, TWIZ, in the Alexa Prize TaskBot Challenge 2022. Our vision, is to build TWIZ bot as an helpful, multimodal, knowledgeable, and engaging assistant that can guide users towards the successful completion of complex manual tasks. To achieve this, we focus our efforts on three main research questions: (1) Humanly-Shaped Conversations, by providing information in a knowledgeable way; (2) Multimodal Stimulus, making use of various modalities including voice, images, and videos; and (3) Zero-shot Conversational Flows, to improve the robustness of the interaction to unseen scenarios. TWIZ is an assistant capable of supporting a wide range of tasks, with several innovative features such as creative cooking, video navigation through voice, and the robust TWIZ-LLM, a Large Language Model trained for dialoguing about complex manual tasks. Given ratings and feedback provided by users, we observed that TWIZ bot is an effective and robust system, capable of guiding users through tasks while providing several multimodal stimuli.
Paper Structure (50 sections, 8 figures, 10 tables)

This paper contains 50 sections, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. TWIZ Bot Modular Architecture
  3. Intent Detection
  4. Dialogue Manager
  5. Dialogue Backbone-Flow
  6. Dialogue Sub-flows
  7. RQ1: Humanly-shaped Conversations
  8. Frictionless Task Suggestions and Search
  9. Creative Cooking: What's in your fridge?
  10. Task Overview: Task Promoter
  11. Evaluation
  12. Task Execution: TWIZ LLM
  13. Conversational Data Augmentation
  14. Conversation Flow
  15. Task Selection
  16. ...and 35 more sections

Figures (8)

  • Figure 1: Architecture diagram of the TWIZ bot.
  • Figure 2: TWIZ's APL screens: welcome screen, search results, "What's in my fridge?", ingredients list, task step, and video dialogue.
  • Figure 3: Video moment retrieval results.
  • Figure 4: TWIZ users can ask a question about a video and navigate to the correct video moment that answers the question. An example interaction can be seen in this https://www.youtube.com/playlist?list=PLC5saXed4eNsebM8C4W5S_BQ9ADEgwH57.
  • Figure 5: TWIZ's 7-Day average rating since the semi-finals for conversations with at least 3 turns.
  • ...and 3 more figures