Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When AI Takes the Wheel: Security Analysis of Framework-Constrained Program Generation

Yue Liu, Zhenchang Xing, Shidong Pan, Chakkrit Tantithamthavorn

TL;DR

The paper investigates security in framework-constrained software generated end-to-end by LLMs, using Chrome extensions as a representative case. It introduces ChromeSecBench, a 140-prompt dataset, and analyzes nine LLMs with CoCo static taint analysis across 10 scenarios and three dimensions to detect dangerous data flows. Findings show alarmingly high vulnerability rates (18%–50%), especially in Authentication & Identity and Cookie Management, with privileged storage data often exposed to untrusted contexts; advanced reasoning models sometimes generate more vulnerabilities, underscoring a gap between coding ability and secure architecture. The work highlights the need for secure AI coding benchmarks, framework-aware training, and automated security checks to bridge the gap between AI-assisted development and secure, policy-compliant software practice.

Abstract

In recent years, the AI wave has grown rapidly in software development. Even novice developers can now design and generate complex framework-constrained software systems based on their high-level requirements with the help of Large Language Models (LLMs). However, when LLMs gradually "take the wheel" of software development, developers may only check whether the program works. They often miss security problems hidden in how the generated programs are implemented. In this work, we investigate the security properties of framework-constrained programs generated by state-of-the-art LLMs. We focus specifically on Chrome extensions due to their complex security model involving multiple privilege boundaries and isolated components. To achieve this, we built ChromeSecBench, a dataset with 140 prompts based on known vulnerable extensions. We used these prompts to instruct nine state-of-the-art LLMs to generate complete Chrome extensions, and then analyzed them for vulnerabilities across three dimensions: scenario types, model differences, and vulnerability categories. Our results show that LLMs produced vulnerable programs at alarmingly high rates (18%-50%), particularly in Authentication & Identity and Cookie Management scenarios (up to 83% and 78% respectively). Most vulnerabilities exposed sensitive browser data like cookies, history, or bookmarks to untrusted code. Interestingly, we found that advanced reasoning models performed worse, generating more vulnerabilities than simpler models. These findings highlight a critical gap between LLMs' coding skills and their ability to write secure framework-constrained programs.

When AI Takes the Wheel: Security Analysis of Framework-Constrained Program Generation

TL;DR

The paper investigates security in framework-constrained software generated end-to-end by LLMs, using Chrome extensions as a representative case. It introduces ChromeSecBench, a 140-prompt dataset, and analyzes nine LLMs with CoCo static taint analysis across 10 scenarios and three dimensions to detect dangerous data flows. Findings show alarmingly high vulnerability rates (18%–50%), especially in Authentication & Identity and Cookie Management, with privileged storage data often exposed to untrusted contexts; advanced reasoning models sometimes generate more vulnerabilities, underscoring a gap between coding ability and secure architecture. The work highlights the need for secure AI coding benchmarks, framework-aware training, and automated security checks to bridge the gap between AI-assisted development and secure, policy-compliant software practice.

Abstract

In recent years, the AI wave has grown rapidly in software development. Even novice developers can now design and generate complex framework-constrained software systems based on their high-level requirements with the help of Large Language Models (LLMs). However, when LLMs gradually "take the wheel" of software development, developers may only check whether the program works. They often miss security problems hidden in how the generated programs are implemented. In this work, we investigate the security properties of framework-constrained programs generated by state-of-the-art LLMs. We focus specifically on Chrome extensions due to their complex security model involving multiple privilege boundaries and isolated components. To achieve this, we built ChromeSecBench, a dataset with 140 prompts based on known vulnerable extensions. We used these prompts to instruct nine state-of-the-art LLMs to generate complete Chrome extensions, and then analyzed them for vulnerabilities across three dimensions: scenario types, model differences, and vulnerability categories. Our results show that LLMs produced vulnerable programs at alarmingly high rates (18%-50%), particularly in Authentication & Identity and Cookie Management scenarios (up to 83% and 78% respectively). Most vulnerabilities exposed sensitive browser data like cookies, history, or bookmarks to untrusted code. Interestingly, we found that advanced reasoning models performed worse, generating more vulnerabilities than simpler models. These findings highlight a critical gap between LLMs' coding skills and their ability to write secure framework-constrained programs.
Paper Structure (24 sections, 4 figures, 5 tables)

This paper contains 24 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. AI for Program Generation
  4. Security Concerns in AI Solutions
  5. Framework-Constrained Extension Security
  6. Experimental Method
  7. Problem Definition
  8. Evaluating Extensions Using Static Analysis
  9. Scenario Design and Implementation
  10. Experimental Setup
  11. Experimental Analysis
  12. Diversity of Scenarios
  13. Diversity of LLMs
  14. Diversity of Vulnerabilities
  15. Discussion
  16. ...and 9 more sections

Figures (4)

  • Figure 1: An Example of Copilot Workspace Generating and Deploying a Chrome Extension
  • Figure 2: Chrome Extension Framework
  • Figure 3: Overview of Our Experimental Workflow
  • Figure 4: Vulnerability Occurrence Across Different Extension Scenarios (darker colors indicate a higher percentage of vulnerable generations per scenario)