LLMs + Security = Trouble

TL;DR

The paper confronts the problem that AI-assisted code generation often yields insecure outputs and that post-hoc vulnerability detection and repair struggle to cover long-tail security bugs. It advocates shifting security guarantees to generation-time constraints, including constrained decoding and diffusion-based generation, to enable secure-by-construction code. It surveys neurosymbolic verification, structured representations, and a broad set of guardrails and evaluation frameworks, highlighting both progress and remaining challenges in achieving robust, scalable security for AI-generated software. The work emphasizes a systems-aware approach to AI security, arguing for end-to-end verification, formal methods, and modular enforcement to reduce reliance on brittle post-hoc checks and human-in-the-loop bottlenecks. Collectively, it provides a taxonomy of methods, evidence across multiple studies, and a practical path toward safer AI-assisted programming in real-world settings.

Abstract

We argue that when it comes to producing secure code with AI, the prevailing "fighting fire with fire" approach -- using probabilistic AI-based checkers or attackers to secure probabilistically generated code -- fails to address the long tail of security bugs. As a result, systems may remain exposed to zero-day vulnerabilities that can be discovered by better-resourced or more persistent adversaries. While neurosymbolic approaches that combine LLMs with formal methods are attractive in principle, we argue that they are difficult to reconcile with the "vibe coding" workflow common in LLM-assisted development: unless the end-to-end verification pipeline is fully automated, developers are repeatedly asked to validate specifications, resolve ambiguities, and adjudicate failures, making the human-in-the-loop a likely point of weakness, compromising secure-by-construction guarantees. In this paper we argue that stronger security guarantees can be obtained by enforcing security constraints during code generation (e.g., via constrained decoding), rather than relying solely on post-hoc detection and repair. This direction is particularly promising for diffusion-style code models, whose approach provides a natural elegant opportunity for modular, hierarchical security enforcement, allowing us to combine lower-latency generation techniques with generating secure-by-construction code.