Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Best Ends by the Best Means: Ethical Concerns in App Reviews

Lauren Olson, Neelam Tjikhoeri, Emitzá Guzmán

TL;DR

The paper investigates ethical concerns in user reviews of mobile apps by constructing a user-informed taxonomy based on Wright's ICT framework and extending it with contemporary concerns. It analyzes 5.86 million reviews and manually labels 3,101 to train and evaluate automated detectors, uncovering that censorship, identity theft, and safety are among the most frequent concerns and that reviews mentioning ethics tend to be longer, more up-voted, and more negatively rated. The study demonstrates high automation potential, achieving strong F1 scores for detecting ethical concerns, internal/external labeling, and multi-class categorization, while noting potential biases from keyword filtering. Practically, the work provides a dataset, models, and actionable guidance for practitioners to systematically incorporate end-user ethical concerns into software design, testing, and governance, thereby supporting proactive risk management and compliance.

Abstract

This work analyzes ethical concerns found in users' app store reviews. We performed this study because ethical concerns in mobile applications (apps) are widespread, pose severe threats to end users and society, and lack systematic analysis and methods for detection and classification. In addition, app store reviews allow practitioners to collect users' perspectives, crucial for identifying software flaws, from a geographically distributed and large-scale audience. For our analysis, we collected five million user reviews, developed a set of ethical concerns representative of user preferences, and manually labeled a sample of these reviews. We found that (1) users highly report ethical concerns about censorship, identity theft, and safety (2) user reviews with ethical concerns are longer, more popular, and lowly rated, and (3) there is high automation potential for the classification and filtering of these reviews. Our results highlight the relevance of using app store reviews for the systematic consideration of ethical concerns during software evolution.

The Best Ends by the Best Means: Ethical Concerns in App Reviews

TL;DR

The paper investigates ethical concerns in user reviews of mobile apps by constructing a user-informed taxonomy based on Wright's ICT framework and extending it with contemporary concerns. It analyzes 5.86 million reviews and manually labels 3,101 to train and evaluate automated detectors, uncovering that censorship, identity theft, and safety are among the most frequent concerns and that reviews mentioning ethics tend to be longer, more up-voted, and more negatively rated. The study demonstrates high automation potential, achieving strong F1 scores for detecting ethical concerns, internal/external labeling, and multi-class categorization, while noting potential biases from keyword filtering. Practically, the work provides a dataset, models, and actionable guidance for practitioners to systematically incorporate end-user ethical concerns into software design, testing, and governance, thereby supporting proactive risk management and compliance.

Abstract

This work analyzes ethical concerns found in users' app store reviews. We performed this study because ethical concerns in mobile applications (apps) are widespread, pose severe threats to end users and society, and lack systematic analysis and methods for detection and classification. In addition, app store reviews allow practitioners to collect users' perspectives, crucial for identifying software flaws, from a geographically distributed and large-scale audience. For our analysis, we collected five million user reviews, developed a set of ethical concerns representative of user preferences, and manually labeled a sample of these reviews. We found that (1) users highly report ethical concerns about censorship, identity theft, and safety (2) user reviews with ethical concerns are longer, more popular, and lowly rated, and (3) there is high automation potential for the classification and filtering of these reviews. Our results highlight the relevance of using app store reviews for the systematic consideration of ethical concerns during software evolution.
Paper Structure (44 sections, 10 figures, 7 tables)

This paper contains 44 sections, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Research Method
  3. Initial Set of Ethical Concerns
  4. Internal and External Concerns
  5. Data Collection
  6. Data Sampling
  7. Manual Annotation Setup (RQ1)
  8. Final Set of Keywords
  9. Final Set of Ethical Concerns
  10. Additional Sample Creation
  11. Inter-Rater Agreement
  12. Analyzed Characteristics (RQ2)
  13. Automated Analysis Setup (RQ3)
  14. Results
  15. Types of Ethical Concerns (RQ1)
  16. ...and 29 more sections

Figures (10)

  • Figure 1: Set of Ethical Concerns and Modification of Wright's Framework
  • Figure 2: Manual Annotation Process
  • Figure 3: Frequency of Ethical Concerns in the Final, Labeled Dataset
  • Figure 4: Frequency of Internal and External Labels Among Ethical Concerns
  • Figure 5: Extent of Ethical Concerns among Applications
  • ...and 5 more figures