Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Openness Fails: Lessons from System Safety for Assessing Openness in AI

Tamara Paris, Shalaleh Rismani

TL;DR

The paper addresses the problem that openness in AI is not automatically realized by releasing data, model, or code; context matters. It borrows five lessons from system safety—deployment context, sociotechnical constraints, mental models, organizational dependencies, and maintenance—to evaluate openness at the system level rather than as fixed component properties. Using a fictional dialect-specific LLM scenario, it demonstrates how openness can fail when release conditions do not align with user capabilities, infrastructure, or policy environments. The contribution provides a pathway for more robust, context-aware openness assessments that can guide developers and policymakers toward sustainable, user-centered openness.

Abstract

Most frameworks for assessing the openness of AI systems use narrow criteria such as availability of data, model, code, documentation, and licensing terms. However, to evaluate whether the intended effects of openness - such as democratization and autonomy - are realized, we need a more holistic approach that considers the context of release: who will reuse the system, for what purposes, and under what conditions. To this end, we adapt five lessons from system safety that offer guidance on how openness can be evaluated at the system level.

When Openness Fails: Lessons from System Safety for Assessing Openness in AI

TL;DR

The paper addresses the problem that openness in AI is not automatically realized by releasing data, model, or code; context matters. It borrows five lessons from system safety—deployment context, sociotechnical constraints, mental models, organizational dependencies, and maintenance—to evaluate openness at the system level rather than as fixed component properties. Using a fictional dialect-specific LLM scenario, it demonstrates how openness can fail when release conditions do not align with user capabilities, infrastructure, or policy environments. The contribution provides a pathway for more robust, context-aware openness assessments that can guide developers and policymakers toward sustainable, user-centered openness.

Abstract

Most frameworks for assessing the openness of AI systems use narrow criteria such as availability of data, model, code, documentation, and licensing terms. However, to evaluate whether the intended effects of openness - such as democratization and autonomy - are realized, we need a more holistic approach that considers the context of release: who will reuse the system, for what purposes, and under what conditions. To this end, we adapt five lessons from system safety that offer guidance on how openness can be evaluated at the system level.

Paper Structure

This paper contains 6 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Failure of AI Openness
  3. The Parallel between System Safety and AI Openness
  4. System Safety Insights for AI Openness
  5. Example Scenario
  6. Lessons from system safety

Figures (2)

  • Figure 1: Illustration of the fictional example scenario. Each part of the diagram is associated with the lesson number it corresponds to.
  • Figure 2: Examples of hazards that can lead to a failure of openness for each of the five lessons. The lessons are indicated by the numbers and colors.