Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications

Luis Morales-Navarro, Yasmin B. Kafai, Vedya Konda, Danaë Metaxa

TL;DR

This paper investigates whether youth can function as peer auditors of ML-powered applications. It builds a two-week, in-person workshop where 13 youths design and audit peer projects, followed by pre/post clinical interviews to assess changes in auditing reasoning. The results show that after the workshop, all participants identified potential biases and data/model design issues, discussed algorithmic justice, and developed ideas to improve their own models, providing empirical support for youth-based algorithm auditing. The work contributes a conceptualization of youth as auditors within child-computer interaction and demonstrates potential benefits for computational empowerment and learning in youth communities.

Abstract

As artificial intelligence/machine learning (AI/ML) applications become more pervasive in youth lives, supporting them to interact, design, and evaluate applications is crucial. This paper positions youth as auditors of their peers' ML-powered applications to better understand algorithmic systems' opaque inner workings and external impacts. In a two-week workshop, 13 youth (ages 14-15) designed and audited ML-powered applications. We analyzed pre/post clinical interviews in which youth were presented with auditing tasks. The analyses show that after the workshop all youth identified algorithmic biases and inferred dataset and model design issues. Youth also discussed algorithmic justice issues and ML model improvements. Furthermore, youth reflected that auditing provided them new perspectives on model functionality and ideas to improve their own models. This work contributes (1) a conceptualization of algorithm auditing for youth; and (2) empirical evidence of the potential benefits of auditing. We discuss potential uses of algorithm auditing in learning and child-computer interaction research.

Youth as Peer Auditors: Engaging Teenagers with Algorithm Auditing of Machine Learning Applications

TL;DR

This paper investigates whether youth can function as peer auditors of ML-powered applications. It builds a two-week, in-person workshop where 13 youths design and audit peer projects, followed by pre/post clinical interviews to assess changes in auditing reasoning. The results show that after the workshop, all participants identified potential biases and data/model design issues, discussed algorithmic justice, and developed ideas to improve their own models, providing empirical support for youth-based algorithm auditing. The work contributes a conceptualization of youth as auditors within child-computer interaction and demonstrates potential benefits for computational empowerment and learning in youth communities.

Abstract

As artificial intelligence/machine learning (AI/ML) applications become more pervasive in youth lives, supporting them to interact, design, and evaluate applications is crucial. This paper positions youth as auditors of their peers' ML-powered applications to better understand algorithmic systems' opaque inner workings and external impacts. In a two-week workshop, 13 youth (ages 14-15) designed and audited ML-powered applications. We analyzed pre/post clinical interviews in which youth were presented with auditing tasks. The analyses show that after the workshop all youth identified algorithmic biases and inferred dataset and model design issues. Youth also discussed algorithmic justice issues and ML model improvements. Furthermore, youth reflected that auditing provided them new perspectives on model functionality and ideas to improve their own models. This work contributes (1) a conceptualization of algorithm auditing for youth; and (2) empirical evidence of the potential benefits of auditing. We discuss potential uses of algorithm auditing in learning and child-computer interaction research.
Paper Structure (28 sections, 4 figures, 1 table)

This paper contains 28 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Youth as testers and evaluators
  4. Youth as auditors of AI/ML applications
  5. Youth’s perspectives towards algorithmic justice
  6. A new role for youth
  7. Youth as peer auditors
  8. Methods
  9. Participants
  10. Workshop activities
  11. Interview design and data collection
  12. Analysis
  13. Findings
  14. How did youth’s identification of potential algorithmic biases and harm change?
  15. Identifying algorithmic biases
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The process of peer auditing involves: (1) youth creating factsheets for the projects they designed, (2) exchanging projects with peers for auditing, which involves iteratively querying the system and comparing auditor-expected classification outputs to system classification outputs, and (3) writing audit reports.
  • Figure 2: Auditing youth's final projects. (A) A participant audits Jackie Star and Emily's drawing game. (B) A participant audits Andrés sports game. (C & D) A participant audits Iván and Walter's fighting game.
  • Figure 3: Faulty classifiers used in pre/post auditing tasks included a berry classifier, a pet classifier, a drawing tool classifier, and a sea animal classifier. The figure shows cases in which the classifiers did not work as expected.
  • Figure 4: Images generated by DALL-E mini for the following prompts: "beautiful woman," "bad student", "teacher", "librarian", "scientist", and "police officer".