Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation

Chengran Yang, Zichao Wei, Heminghao Deng, Jinfeng Jiang, Zhensu Sun, Ting Zhang, Tianyi Wu, Ming Wen, David Lo

TL;DR

This work reframes code generation as a dynamic drafting process by introducing Stream of Revision, which integrates just-in-time, in-pass edits through a Virtual Cursor and executable revision tokens. A deterministic renderer then applies the edits, enabling secure code generation without external tools or multi-turn cycles. The approach achieves strong security on CyberSecEval 2 while preserving functionality, with data-efficient alignment from real CVE patches and favorable efficiency comparisons to post-hoc repair methods. Additionally, SoR demonstrates zero-shot transfer to unseen languages and complements weight-level alignment methods like ProSec, offering practical improvements for trustworthy code generation in real-world software tasks.

Abstract

Large Language Model (LLM) based code generation is predominantly formulated as a strictly monotonic process, appending tokens linearly to an immutable prefix. This formulation contrasts to the cognitive process of programming, which is inherently interleaved with forward generation and on-the-fly revision. While prior works attempt to introduce revision via post-hoc agents or external static tools, they either suffer from high latency or fail to leverage the model's intrinsic semantic reasoning. In this paper, we propose Stream of Revision, a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory by leveraging model's intrinsic capabilities. We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass. By internalizing the revision loop, our framework Stream of Revision allows the model to activate its latent capabilities just-in-time without external dependencies. Empirical results on secure code generation show that Stream of Revision significantly reduces vulnerabilities with minimal inference overhead.

Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation

TL;DR

This work reframes code generation as a dynamic drafting process by introducing Stream of Revision, which integrates just-in-time, in-pass edits through a Virtual Cursor and executable revision tokens. A deterministic renderer then applies the edits, enabling secure code generation without external tools or multi-turn cycles. The approach achieves strong security on CyberSecEval 2 while preserving functionality, with data-efficient alignment from real CVE patches and favorable efficiency comparisons to post-hoc repair methods. Additionally, SoR demonstrates zero-shot transfer to unseen languages and complements weight-level alignment methods like ProSec, offering practical improvements for trustworthy code generation in real-world software tasks.

Abstract

Large Language Model (LLM) based code generation is predominantly formulated as a strictly monotonic process, appending tokens linearly to an immutable prefix. This formulation contrasts to the cognitive process of programming, which is inherently interleaved with forward generation and on-the-fly revision. While prior works attempt to introduce revision via post-hoc agents or external static tools, they either suffer from high latency or fail to leverage the model's intrinsic semantic reasoning. In this paper, we propose Stream of Revision, a paradigm shift that elevates code generation from a monotonic stream to a dynamic, self-correcting trajectory by leveraging model's intrinsic capabilities. We introduce specific action tokens that enable the model to seamlessly backtrack and edit its own history within a single forward pass. By internalizing the revision loop, our framework Stream of Revision allows the model to activate its latent capabilities just-in-time without external dependencies. Empirical results on secure code generation show that Stream of Revision significantly reduces vulnerabilities with minimal inference overhead.
Paper Structure (77 sections, 20 equations, 5 figures, 5 tables)

This paper contains 77 sections, 20 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Task Definition: Secure Code Generation
  4. The Autoregressive Bottleneck
  5. The Immutability Constraint.
  6. The Commitment Trap.
  7. Methodology
  8. Representational Primitives: The Virtual Cursor
  9. I. Vulnerability Detection as Action Trigger ($\tau_{\text{trig}}$).
  10. II. Vulnerability Localization via Content-Addressable Scope.
  11. III. Vulnerability Repair via Explicit Patch Synthesis.
  12. The Deterministic Renderer
  13. Training Strategy and Optimization
  14. Semantic Embedding Initialization
  15. Optimization Objectives
  16. ...and 62 more sections

Figures (5)

  • Figure 1: Stream of Generation vs. Stream of Revision. Conventional code generation treats generation as a linear stream of token appending, lacking the ability to revise earlier tokens. In contrast, our proposed Stream of Revision framework introduces action tokens that enable dynamic backtracking and in-place editing within a single pass.
  • Figure 2: Overview of Stream of Revision for alignment data construction and single pass inference. Top: from real world CVE pairs, we filter, extract code diffs, and linearize the change into a revision trajectory with an revision trigger, a localized vulnerable span, and a patch span. Bottom: during autoregressive decoding, the model can emit a trigger token to start a revision episode, localize a vulnerable span under constrained decoding, and then generate a patch span. A deterministic renderer applies the patch atomically to the user facing buffer. Red highlights vulnerable code, green highlights patched code.
  • Figure 3: Impact of Training Data Scale.Blue bars (Left Axis) denote Security Pass Rate (SPR), Orange bars (Right Axis) denote Avg. Inference Tokens. Comparing the hatched bars to solid bars shows that adding more data yields negligible security gains but incurs higher inference costs due to more revisions.
  • Figure 4: Per-Category Secure Patch Rate (SPR) on Top-10 CWEs. Comparison between the vanilla base model and Stream of Revision.
  • Figure 5: Case Study of Stream of Revision in Action.