Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp

TL;DR

The paper tackles the challenge of scalable UX analysis by combining a supervised, multi-label comment classifier with SBERT/fastText embeddings, an extreme gradient boosting classifier, and a human-in-the-loop workflow to maintain label accuracy. It extends this with GenAI-driven, category-aligned summaries and a validation pipeline to keep summaries faithful to source data. Crucially, it investigates how sentiment in open-ended feedback relates to formal satisfaction metrics, finding that positive sentiment reliably signals higher satisfaction while negative sentiment is a weak proxy for dissatisfaction, highlighting the need for explicit satisfaction measures. The approach yields practical, scalable insights for product teams, including dashboards, drift-aware topic updates, and a pathway toward automating actionable feedback while acknowledging limitations of sentiment as a stand-alone metric.

Abstract

In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.

Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

TL;DR

The paper tackles the challenge of scalable UX analysis by combining a supervised, multi-label comment classifier with SBERT/fastText embeddings, an extreme gradient boosting classifier, and a human-in-the-loop workflow to maintain label accuracy. It extends this with GenAI-driven, category-aligned summaries and a validation pipeline to keep summaries faithful to source data. Crucially, it investigates how sentiment in open-ended feedback relates to formal satisfaction metrics, finding that positive sentiment reliably signals higher satisfaction while negative sentiment is a weak proxy for dissatisfaction, highlighting the need for explicit satisfaction measures. The approach yields practical, scalable insights for product teams, including dashboards, drift-aware topic updates, and a pathway toward automating actionable feedback while acknowledging limitations of sentiment as a stand-alone metric.

Abstract

In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.
Paper Structure (16 sections, 2 figures, 4 tables)

This paper contains 16 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Developing a Multi-Label Comment Classification Model
  3. Deriving Topic Labels
  4. Creating a Training Dataset
  5. Generating Comment Embeddings
  6. Specifying a Multi-Label Classification Model
  7. Hyperparameter Tuning
  8. Incremental Learning from Human Feedback
  9. Generating Comment Summaries
  10. Insufficient and Inefficient Results
  11. Our Best Approach
  12. Interpreting Sentiments
  13. A Survey Concerning the Quality of Tutorials
  14. A Survey Measuring the Usability of Applications
  15. What Does This Mean for the Interpretation of Sentiments?
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Cumulative frequency of the tutorial quality score for the three sentiment categories. Respondents providing positive comments show with a few exceptions high tutorial quality scores. On the other hand, negative comments and high tutorial quality scores occur together quite frequently. Thus, a negative comment does not necessarily indicate a low satisfaction with the tutorial.
  • Figure 2: Cumulative frequency of the UX-Lite scores for the three sentiment categories. Respondents that provide positive comments show with a few exceptions high UX-Lite scores. On the other hand, negative comments and high UX-Lite scores occur together quite frequently, i.e. negative comments do not imply a low overall satisfaction of users.