Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An engine not a camera: Measuring performative power of online search

Celestine Mendler-Dünner, Gabriele Carovano, Moritz Hardt

TL;DR

An experiment to measure the performative power of online search providers is designed and executed, using the Google Shopping antitrust investigation as a case study and envisioned to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.

Abstract

The power of digital platforms is at the center of major ongoing policy and regulatory efforts. To advance existing debates, we designed and executed an experiment to measure the performative power of online search providers. Instantiated in our setting, performative power quantifies the ability of a search engine to steer web traffic by rearranging results. To operationalize this definition we developed a browser extension that performs unassuming randomized experiments in the background. These randomized experiments emulate updates to the search algorithm and identify the causal effect of different content arrangements on clicks. Analyzing tens of thousands of clicks, we discuss what our robust quantitative findings say about the power of online search engines, using the Google Shopping antitrust investigation as a case study. More broadly, we envision our work to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.

An engine not a camera: Measuring performative power of online search

TL;DR

An experiment to measure the performative power of online search providers is designed and executed, using the Google Shopping antitrust investigation as a case study and envisioned to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.

Abstract

The power of digital platforms is at the center of major ongoing policy and regulatory efforts. To advance existing debates, we designed and executed an experiment to measure the performative power of online search providers. Instantiated in our setting, performative power quantifies the ability of a search engine to steer web traffic by rearranging results. To operationalize this definition we developed a browser extension that performs unassuming randomized experiments in the background. These randomized experiments emulate updates to the search algorithm and identify the causal effect of different content arrangements on clicks. Analyzing tens of thousands of clicks, we discuss what our robust quantitative findings say about the power of online search engines, using the Google Shopping antitrust investigation as a case study. More broadly, we envision our work to serve as a blueprint for how the recent definition of performative power can help integrate quantitative insights from online experiments with future investigations into the economic power of digital platforms.
Paper Structure (32 sections, 1 theorem, 6 equations, 11 figures, 1 table)

This paper contains 32 sections, 1 theorem, 6 equations, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Preliminaries and related work
  4. Performative power.
  5. The Google Shopping case.
  6. Display effects and behavioral biases.
  7. Measuring the effect of algorithmic updates.
  8. Performativity in online search
  9. The causal effect of arrangement
  10. Algorithmic distortion.
  11. Estimating the performativity gap using an RCT
  12. Powermeter: Experiment in the wild
  13. Browser extension
  14. Technical details.
  15. Backend and data collection.
  16. ...and 17 more sections

Key Result

Theorem 1

Let $\mathrm{PP}$ be the performative power of a search platform defined with respect to a set of arrangements $\mathcal{A}$, a population of search queries $\mathcal{Q}$ performed on the platform, and the outcome variable $z_a(q) = \mathrm{1}\{C_q(a)=c_1\}$. Then, performative power is lower bounde

Figures (11)

  • Figure 1: The ability to influence web traffic through content arrangement. Blue bars show average click probability observed for generic search results in position 1 to 6 on Google search under different counterfactual arrangements; default arrangement (left), swapping results 1 and 2 (middle), swapping results 1 and 3 (right). We provide a detailed discussion in Section \ref{['sec:results']} where we also explore arrangement changes beyond reranking.
  • Figure 2: Illustration of different elements on the Google search website.
  • Figure 3: Click through rate and performativity gap for general search results $c_1$ to $c_6$ under the counterfactual arrangements $a_1$, $a_2$, $a_3$ for Google and the counterfactual arrangement $a_1$ for Bing, compared to the control arrangement $a_0$ (in blue).
  • Figure 4: Effect of arrangement on the click distribution across different element types (generic search results, Ads, boxes), visualized for three different subsets of Google queries.
  • Figure 5: Effect of hiding box and swapping elements on the click probability of generic search results. Statistics are evaluated for the subset of queries for which box is naturally present. The hashed bar shows the click probability under $a_5$ when top content is hidden and the first two elements are swapped. The right table reports the empirical measure of algorithmic distortion for different conducts, extracted form the results in the left figure.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Definition 1: Performative power hardt2022performative
  • Definition 2: Performativity gap
  • Theorem 1: Lowerbound on performative power