Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Saltzer & Schroeder for 2030: Security engineering principles in a world of AI

Nikhil Patnaik, Joseph Hallett, Awais Rashid

TL;DR

The paper addresses the security of AI-generated code by revisiting Saltzer & Schroeder's design principles in the context of a world where large-language models produce software artifacts. It conducts a password-storage case study with ChatGPT, evaluating outputs against Naiakshina's secure password criteria and then against Saltzer & Schroeder principles, with explicit-security prompts improving results to $5/8$ on average and up to $8/8$ in one case. The findings show partial adherence and misapplications of the principles by default, underscoring the need to adapt or extend the principles for AI-assisted development and to train models on secure-design tasks. The work argues for a security-focused trajectory where Saltzer & Schroeder are embedded into prompt design, training datasets, and tooling, shaping practical guidance for secure AI-enabled software by 2030. This has implications for how developers interact with AI tools, how security APIs are designed, and how future LLMs are trained to internalize robust security design.

Abstract

Writing secure code is challenging and so it is expected that, following the release of code-generative AI tools, such as ChatGPT and GitHub Copilot, developers will use these tools to perform security tasks and use security APIs. However, is the code generated by ChatGPT secure? How would the everyday software or security engineer be able to tell? As we approach the next decade we expect a greater adoption of code-generative AI tools and to see developers use them to write secure code. In preparation for this, we need to ensure security-by-design. In this paper, we look back in time to Saltzer & Schroeder's security design principles as they will need to evolve and adapt to the challenges that come with a world of AI-generated code.

Saltzer & Schroeder for 2030: Security engineering principles in a world of AI

TL;DR

The paper addresses the security of AI-generated code by revisiting Saltzer & Schroeder's design principles in the context of a world where large-language models produce software artifacts. It conducts a password-storage case study with ChatGPT, evaluating outputs against Naiakshina's secure password criteria and then against Saltzer & Schroeder principles, with explicit-security prompts improving results to on average and up to in one case. The findings show partial adherence and misapplications of the principles by default, underscoring the need to adapt or extend the principles for AI-assisted development and to train models on secure-design tasks. The work argues for a security-focused trajectory where Saltzer & Schroeder are embedded into prompt design, training datasets, and tooling, shaping practical guidance for secure AI-enabled software by 2030. This has implications for how developers interact with AI tools, how security APIs are designed, and how future LLMs are trained to internalize robust security design.

Abstract

Writing secure code is challenging and so it is expected that, following the release of code-generative AI tools, such as ChatGPT and GitHub Copilot, developers will use these tools to perform security tasks and use security APIs. However, is the code generated by ChatGPT secure? How would the everyday software or security engineer be able to tell? As we approach the next decade we expect a greater adoption of code-generative AI tools and to see developers use them to write secure code. In preparation for this, we need to ensure security-by-design. In this paper, we look back in time to Saltzer & Schroeder's security design principles as they will need to evolve and adapt to the challenges that come with a world of AI-generated code.
Paper Structure (10 sections, 1 figure)

This paper contains 10 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. General insights
  4. Security implications of using AI models
  5. Case Study: Secure Password Storage with ChatGPT
  6. ChatGPT Without Explicit Security Requirements
  7. ChatGPT with Explicit Security Requirements
  8. ChatGPT and Saltzer & Schroeder
  9. Discussion
  10. Research Agenda 2030

Figures (1)

  • Figure 1: Naiakshina's end-user password storage assessment criteria naiakshina2017developers, copied verbatim. A score $\mathbf{\geq 6}$ indicates industrial best practice.