Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applications

TL;DR

A key finding of the case study is that involving domain experts increases their interest and trust in the final NLP application, and the proposed method efficiently considers data- and user-driven opportunities and constraints, which can be crucial for developing NLP applications.

Abstract

User experience (UX) is a part of human-computer interaction (HCI) research and focuses on increasing intuitiveness, transparency, simplicity, and trust for the system users. Most UX research for machine learning (ML) or natural language processing (NLP) focuses on a data-driven methodology. It engages domain users mainly for usability evaluation. Moreover, more typical UX methods tailor the systems towards user usability, unlike learning about the user needs first. This paper proposes a new methodology for integrating generative UX research into developing domain NLP applications. Generative UX research employs domain users at the initial stages of prototype development, i.e., ideation and concept evaluation, and the last stage for evaluating system usefulness and user utility. The methodology emerged from and is evaluated on a case study about the full-cycle prototype development of a domain-specific semantic search for daily operations in the process industry. A key finding of our case study is that involving domain experts increases their interest and trust in the final NLP application. The combined UX+NLP research of the proposed method efficiently considers data- and user-driven opportunities and constraints, which can be crucial for developing NLP applications.

Figures (15)

• Figure 1: A timeline for the proposed methodology of generative UX research for prototyping domain NLP applications used in our case study. The methodology consists of three stages: exploration (data exploration and generative research), prototyping (development of the prototype and evaluation (assessment of accuracy and user utility). The phases show the approximate share of the workload of the UX and NLP teams. For the majority of the phases, both teams need to be involved. Especially during the generative research phase, an equal involvement of the two teams enables deep domain understanding.
• Figure 2: A simplified example of a record in a logging system for the daily operations in the process industry. A record is semi-structured, i.e., it has structured attributes of the records and unstructured text descriptions. The domain language contains professional terms, abbreviations, shortenings, chemical formulas, various IDs (e.g., order IDs), and plant machinery and equipment inventory codes. The example is in English for better readability.
• Figure 3: Quantitative analysis of the domain text records of a logging system. Left: A histogram of the length of text records shows that most records are very short, i.e., smaller than 200 chars. Right: The domain vocabulary includes words and many codes of functional locations (i.e., machinery) that act like nouns. Additionally, the vocabulary contains a lot of digit-containing terms that refer to IDs or numerical values, e.g., measurements.
• Figure 4: Analysis of the domain language shows a significant difference to a general domain (Wikipedia): t-SNE projection of the vectorized domain and general vocabulary clearly shows spots with no overlap between the most frequent terms. Left plot: 1000 general and domain most frequent terms. Right plot: 1000 general and 200 domain most frequent terms. In the list of frequent domain terms, "Product_#" is an anonymized version of the real product names.
• Figure 5: A generalization of a shift from a shift leader resulting from our interviews that acts as a domain model for our user study. The context inquiries identified three phases of a shift and the main tasks and events that may occur during the day. We provide a short description and outline the challenges of the steps in all phases, which we can turn into opportunities for an NLP application.