Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

To Err is AI : A Case Study Informing LLM Flaw Reporting Practices

Sean McGregor, Allyson Ettinger, Nick Judd, Paul Albee, Liwei Jiang, Kavel Rao, Will Smith, Shayne Longpre, Avijit Ghosh, Christopher Fiorelli, Michelle Hoang, Sven Cattell, Nouha Dziri

TL;DR

A collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs), are presented.

Abstract

In August of 2024, 495 hackers generated evaluations in an open-ended bug bounty targeting the Open Language Model (OLMo) from The Allen Institute for AI. A vendor panel staffed by representatives of OLMo's safety program adjudicated changes to OLMo's documentation and awarded cash bounties to participants who successfully demonstrated a need for public disclosure clarifying the intent, capacities, and hazards of model deployment. This paper presents a collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs). These include best practices for safety reporting processes, their artifacts, and safety program staffing.

To Err is AI : A Case Study Informing LLM Flaw Reporting Practices

TL;DR

A collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs), are presented.

Abstract

In August of 2024, 495 hackers generated evaluations in an open-ended bug bounty targeting the Open Language Model (OLMo) from The Allen Institute for AI. A vendor panel staffed by representatives of OLMo's safety program adjudicated changes to OLMo's documentation and awarded cash bounties to participants who successfully demonstrated a need for public disclosure clarifying the intent, capacities, and hazards of model deployment. This paper presents a collection of lessons learned, illustrative of flaw reporting best practices intended to reduce the likelihood of incidents and produce safer large language models (LLMs). These include best practices for safety reporting processes, their artifacts, and safety program staffing.

Paper Structure

This paper contains 11 sections, 5 figures.

Table of Contents

  1. Introduction
  2. LLM Vendor Challenges
  3. Challenge 1. Tooling support for flaw reporting
  4. Challenge 2. Adjudication Workload
  5. Challenge 3. LLM Documentation Practices
  6. Challenge 4. Identifying the target
  7. Challenge 5. Adjudication Decisions
  8. Denylisting vs. Allowlisting.
  9. Challenge 6. Adjudication Expertise
  10. Discussion
  11. Acknowledgments

Figures (5)

  • Figure 1: Generative Red Team 2 signage greeting prospective participants as they wander through the Las Vegas Convention Center.
  • Figure 2: A closeup of the vendor adjudication table. DEF CON prohibits large group photography. The adjudication team was drawn from The Allen Institute for AI and the UL Research Institutes, and was supported by people from Dreadnode, Bugcrowd, and volunteers from the AI Village. The adjudicators were researchers who work in the areas of LLM research, AI safety, computer security, law, and other areas of AI research.
  • Figure 3: The Crucible user interface displaying the user input and model response.
  • Figure 4: The user interface displaying the metadata associated with a flaw report submitted from Crucible to Bugcrowd.
  • Figure 5: The re-prompting strategy employed when handing off from WildGuard to OLMo.