Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents

TL;DR

The paper tackles the safety challenges of autonomous FM-based agents by proposing a Swiss Cheese-inspired multi-layer guardrail approach. It employs a systematic literature review to derive a two-part contribution: a taxonomy of 14 guardrail quality attributes and a set of design options, and a reference architecture that integrates external environment, agent components, guardrails, and AgentOps. By validating across 32 studies, the work provides concrete guidance for AI-safety-by-design, enabling robust, customizable, and auditable runtime protection across prompts, intermediate steps, and final outputs as well as across agent artifacts. The framework aims to address the intrinsic autonomy and non-determinism of FM-based agents, offering a practical blueprint for researchers and practitioners to implement multi-layered, architecture-aware safety controls in real-world systems.

Abstract

Foundation Model (FM)-based agents are revolutionizing application development across various domains. However, their rapidly growing capabilities and autonomy have raised significant concerns about AI safety. Researchers are exploring better ways to design guardrails to ensure that the runtime behavior of FM-based agents remains within specific boundaries. Nevertheless, designing effective runtime guardrails is challenging due to the agents' autonomous and non-deterministic behavior. The involvement of multiple pipeline stages and agent artifacts, such as goals, plans, tools, at runtime further complicates these issues. Addressing these challenges at runtime requires multi-layered guardrails that operate effectively at various levels of the agent architecture. Therefore, in this paper, based on the results of a systematic literature review, we present a comprehensive taxonomy of runtime guardrails for FM-based agents to identify the key quality attributes for guardrails and design dimensions. Inspired by the Swiss Cheese Model, we also propose a reference architecture for designing multi-layered runtime guardrails for FM-based agents, which includes three dimensions: quality attributes, pipelines, and artifacts. The proposed taxonomy and reference architecture provide concrete and robust guidance for researchers and practitioners to build AI-safety-by-design from a software architecture perspective.

Figures (3)

• Figure 1: Methodology
• Figure 2: Taxonomy of multi-layered runtime guardrails for FM-based agents.
• Figure 3: Reference architecture for multi-layered guardrails of FM-based agents.