Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination

Amit Datta, Michael Carl Tschantz, Anupam Datta

TL;DR

This work introduces AdFisher, an automated framework for conducting randomized, controlled browser experiments to study how user behavior, Google Ad Settings, and served ads interact in a black-box advertising ecosystem. By combining blocking, permutation-based significance testing, and machine-learning driven test-statistic selection, the authors demonstrate both opacity (ads change without reflecting in Ad Settings) and discrimination (gender affecting job-ad targeting), while also showing evidence of user-influenced ad choice. Across 21 experiments with ~17k agents and over 600k ads, they provide statistically rigorous evidence of these phenomena and discuss the limitations of attributing blame in such a complex system. The results motivate further scrutiny by regulators and platform operators and establish AdFisher as a scalable tool for auditing personalized advertising practices.

Abstract

To partly address people's concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google's ads, and Ad Settings interact. AdFisher can run browser-based experiments and analyze data using machine learning and significance tests. Our tool uses a rigorous experimental design and statistical analysis to ensure the statistical soundness of our results. We use AdFisher to find that the Ad Settings was opaque about some features of a user's profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse changed the ads shown but not the settings page. We also found that setting the gender to female resulted in getting fewer instances of an ad related to high paying jobs than setting it to male. We cannot determine who caused these findings due to our limited visibility into the ad ecosystem, which includes Google, advertisers, websites, and users. Nevertheless, these results can form the starting point for deeper investigations by either the companies themselves or by regulatory bodies.

Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination

TL;DR

This work introduces AdFisher, an automated framework for conducting randomized, controlled browser experiments to study how user behavior, Google Ad Settings, and served ads interact in a black-box advertising ecosystem. By combining blocking, permutation-based significance testing, and machine-learning driven test-statistic selection, the authors demonstrate both opacity (ads change without reflecting in Ad Settings) and discrimination (gender affecting job-ad targeting), while also showing evidence of user-influenced ad choice. Across 21 experiments with ~17k agents and over 600k ads, they provide statistically rigorous evidence of these phenomena and discuss the limitations of attributing blame in such a complex system. The results motivate further scrutiny by regulators and platform operators and establish AdFisher as a scalable tool for auditing personalized advertising practices.

Abstract

To partly address people's concerns over web tracking, Google has created the Ad Settings webpage to provide information about and some choice over the profiles Google creates on users. We present AdFisher, an automated tool that explores how user behaviors, Google's ads, and Ad Settings interact. AdFisher can run browser-based experiments and analyze data using machine learning and significance tests. Our tool uses a rigorous experimental design and statistical analysis to ensure the statistical soundness of our results. We use AdFisher to find that the Ad Settings was opaque about some features of a user's profile, that it does provide some choice on ads, and that these choices can lead to seemingly discriminatory ads. In particular, we found that visiting webpages associated with substance abuse changed the ads shown but not the settings page. We also found that setting the gender to female resulted in getting fewer instances of an ad related to high paying jobs than setting it to male. We cannot determine who caused these findings due to our limited visibility into the ad ecosystem, which includes Google, advertisers, websites, and users. Nevertheless, these results can form the starting point for deeper investigations by either the companies themselves or by regulatory bodies.

Paper Structure

This paper contains 40 sections, 5 equations, 5 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Problem and Overview
  3. Motivating Experiments
  4. Contributions
  5. Contents
  6. Prior Work
  7. Privacy Properties
  8. Discrimination
  9. Transparency
  10. Choice
  11. Methodology
  12. Background: Significance Testing
  13. Blocking
  14. Selecting Test Statistics
  15. Avoiding Pitfalls
  16. ...and 25 more sections

Figures (5)

  • Figure 1: Screenshot of Google's Ad Settings webpage
  • Figure 2: Experimental setup to carry out significance testing on eight browser agents comparing the effects of two treatments. Each agent is randomly assigned a treatment which specifies what actions to perform on the web. After these actions are complete, they collect measurements which are used for significance testing.
  • Figure 3: Our experimental setup with training and testing blocks. Measurements from the training blocks are used to build a classifier. The trained classifier is used to compute the test statistic on the measurements from the testing blocks for significance testing.
  • Figure 4: Screenshot of an ad with the top URL+title for identifying agents that visited webpages associated with substance abuse
  • Figure 5: For each interest selected for the agents that visited webpages associated with disabilities, the number of agents with that interest selected