Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Online Laboratory: Conducting Experiments in a Real Labor Market

John J. Horton, David G. Rand, Richard J. Zeckhauser

TL;DR

The paper argues that online labor markets provide a practical, valid platform for conducting randomized controlled trials with large, diverse samples at a fraction of traditional costs. It demonstrates this by replicating classic framing, social preferences, and priming effects, plus a labor-supply experiment, all conducted on MTurk with strong internal validity and low costs. The authors discuss threats to validity, strategies to preserve SUTVA and manage attrition, and how external validity depends on theoretical framing, suggesting online studies can complement and extend traditional methods. They advocate for improved software tools, norms for openness and replication, and a broader role for online experimentation in the social sciences.

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid---both internally and externally---as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

The Online Laboratory: Conducting Experiments in a Real Labor Market

TL;DR

The paper argues that online labor markets provide a practical, valid platform for conducting randomized controlled trials with large, diverse samples at a fraction of traditional costs. It demonstrates this by replicating classic framing, social preferences, and priming effects, plus a labor-supply experiment, all conducted on MTurk with strong internal validity and low costs. The authors discuss threats to validity, strategies to preserve SUTVA and manage attrition, and how external validity depends on theoretical framing, suggesting online studies can complement and extend traditional methods. They advocate for improved software tools, norms for openness and replication, and a broader role for online experimentation in the social sciences.

Abstract

Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid---both internally and externally---as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.

Paper Structure

This paper contains 28 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Online Labor Markets and Experimentation
  3. The advantages of recruiting from labor markets
  4. Obtaining control and promoting trust
  5. Experiments in MTurk
  6. Experiments in the Online Laboratory
  7. Existing research
  8. Replication: Framing
  9. Replication: Social preferences
  10. Replication: Priming
  11. Replication: Labor supply on the extensive margin
  12. Summary
  13. Internal Validity
  14. Unique and independent observations
  15. Appropriate assignment
  16. ...and 13 more sections

Figures (2)

  • Figure 1: Reading a religious passage significantly increases prisoner's dilemma cooperation among those who believe in God, but not among non-believers.
  • Figure 2: Self-reported motivation for working on Amazon Mechanical Turk (row) cross-tabulated with self-reported country (column) for 302 workers/subjects.