Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Signed, Sealed,... Confused: Exploring the Understandability and Severity of Policy Documents

Shikha Soneji, Sourav Panda, Sameer Neve, Jonathan Dodge

TL;DR

This study investigates how users understand and judge the severity of ToS;DR policy concepts. Using an online crowd-sourced survey, it quantifies understandability and severity for five random cases, finds generally high understandability but consistent bias toward service-provider advantages in severity, and shows substantial case-by-case variability. Rewrites by participants indicate room to improve labeling, while correlations between understandability and severity suggest that clearer explanations can sharpen user judgment of privacy harms. The work highlights the value of user-centered policy presentation and points to educational interventions and design refinements to promote more informed consent and trust in digital services.

Abstract

In general, Terms of Service (ToS) and other policy documents are verbose and full of legal jargon, which poses challenges for users to understand. To improve user accessibility and transparency, the "Terms of Service; Didn't Read" (ToS;DR) project condenses intricate legal terminology into summaries and overall grades for the website's policy documents. Nevertheless, uncertainties remain about whether users could truly grasp the implications of simplified presentations. We conducted an online survey to assess the perceived understandability and severity of randomly chosen cases from the ToS;DR taxonomy. Preliminary results indicate that, although most users report understanding the cases, they find a bias towards service providers in about two-thirds of the cases. The findings of our study emphasize the necessity of prioritizing user-centric policy formulation. This study has the potential to reveal the extent of information imbalance in digital services and promote more well-informed user consent.

Signed, Sealed,... Confused: Exploring the Understandability and Severity of Policy Documents

TL;DR

This study investigates how users understand and judge the severity of ToS;DR policy concepts. Using an online crowd-sourced survey, it quantifies understandability and severity for five random cases, finds generally high understandability but consistent bias toward service-provider advantages in severity, and shows substantial case-by-case variability. Rewrites by participants indicate room to improve labeling, while correlations between understandability and severity suggest that clearer explanations can sharpen user judgment of privacy harms. The work highlights the value of user-centered policy presentation and points to educational interventions and design refinements to promote more informed consent and trust in digital services.

Abstract

In general, Terms of Service (ToS) and other policy documents are verbose and full of legal jargon, which poses challenges for users to understand. To improve user accessibility and transparency, the "Terms of Service; Didn't Read" (ToS;DR) project condenses intricate legal terminology into summaries and overall grades for the website's policy documents. Nevertheless, uncertainties remain about whether users could truly grasp the implications of simplified presentations. We conducted an online survey to assess the perceived understandability and severity of randomly chosen cases from the ToS;DR taxonomy. Preliminary results indicate that, although most users report understanding the cases, they find a bias towards service providers in about two-thirds of the cases. The findings of our study emphasize the necessity of prioritizing user-centric policy formulation. This study has the potential to reveal the extent of information imbalance in digital services and promote more well-informed user consent.

Paper Structure

This paper contains 45 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related work
  3. The role of social media apps
  4. User's knowledge and attitudes about data collection practices
  5. Users’ Opinions on Annotated Policies
  6. Methodology
  7. Demographic and Attitude Information
  8. Analysis
  9. Results RQ1 - What does the understandability data reveal about participants and privacy concepts?
  10. Which privacy concepts received low understandability ratings?
  11. Which privacy concepts exhibited low consensus about understandability?
  12. How did participants rewrite case labels for low understandability concepts?
  13. Implications
  14. Results RQ2 - What does the severity data reveal about participants and privacy concepts?
  15. Which privacy concepts received high severity ratings?
  16. ...and 30 more sections

Figures (5)

  • Figure 1: Scatterplot of Average Severity (AVG_S) vs Average Understandability (AVG_UA).
  • Figure 2: Relationship between average Understandability scores and average Severity scores. To read the figure, when the understandability score was 1, the mean severity score was 4, and when the severity score was 1, the mean understandability score was about 8, and so on. Note that as the understandability score increases from 1 to 10, the average severity score correspondingly rises from 4 to 6.3.
  • Figure 3: Box Plot for Understandability and Severity scores for different age groups
  • Figure 4: No. of participants facing negative experiences
  • Figure 5: Percentage of Participants facing multiple negative experiences