Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models

Mikio Nakano, Hironori Takeuchi, Kazunori Komatani

TL;DR

This paper proposes a methodology for identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems.

Abstract

This paper proposes a methodology for identifying evaluation items for practical dialogue systems. Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. So far, there has been no methodology for identifying these evaluation items. We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.

A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models

TL;DR

This paper proposes a methodology for identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems.

Abstract

This paper proposes a methodology for identifying evaluation items for practical dialogue systems. Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. So far, there has been no methodology for identifying these evaluation items. We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.
Paper Structure (24 sections, 7 figures, 3 tables)

This paper contains 24 sections, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Evaluating Dialogue Systems
  4. Business-IT Alignment Model
  5. Proposed Methodology
  6. Overview
  7. Generic Model for Business-Dialogue System Alignment
  8. Generic Model of Values
  9. User value
  10. Quality value
  11. Business value
  12. Generic Model of Risks
  13. Generic Model of Costs
  14. Generic Model of Dialogue Systems
  15. Creating a Business-Dialogue System Alignment Model and Identifying Evaluation Items
  16. ...and 9 more sections

Figures (7)

  • Figure 1: Relationships among the business-IT and business-DS alignment models.
  • Figure 2: Generic model of values.
  • Figure 3: Generic model of risks. Not satisfying the AI principles causes risks.
  • Figure 4: Generic model of costs.
  • Figure 5: Generic model of dialogue systems. A solid line without direction denotes a general relationship.
  • ...and 2 more figures