Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PICO: Secure Transformers via Robust Prompt Isolation and Cybersecurity Oversight

Ben Goertzel, Paulos Yibelo

TL;DR

PICO addresses prompt injection in transformers by enforcing a principled invariant through dual-input channels: a frozen, immutable system-prompt stream and a trainable user-input stream. A gated fusion mechanism, enhanced by a Security Expert Agent and a Cybersecurity Knowledge Graph, dynamically biases toward the trusted system context to preserve safe, instruction-compliant outputs. The work provides formal guarantees on adversarial invariance and utility under benign inputs, and outlines concrete architectural realizations, training strategies, and case studies against policy-puppetry. While full training from scratch yields strongest security, the paper also presents an efficient fine-tuning pathway that leverages a pretrained model with dual streams and auxiliary security modules for practical deployment. Collectively, PICO integrates architectural isolation, multi-signal gating, and reinforcement learning to offer robust, end-to-end defenses with clear theoretical and practical benefits for secure LLM use in adversarial settings.

Abstract

We propose a robust transformer architecture designed to prevent prompt injection attacks and ensure secure, reliable response generation. Our PICO (Prompt Isolation and Cybersecurity Oversight) framework structurally separates trusted system instructions from untrusted user inputs through dual channels that are processed independently and merged only by a controlled, gated fusion mechanism. In addition, we integrate a specialized Security Expert Agent within a Mixture-of-Experts (MoE) framework and incorporate a Cybersecurity Knowledge Graph (CKG) to supply domain-specific reasoning. Our training design further ensures that the system prompt branch remains immutable while the rest of the network learns to handle adversarial inputs safely. This PICO framework is presented via a general mathematical formulation, then elaborated in terms of the specifics of transformer architecture, and fleshed out via hypothetical case studies including Policy Puppetry attacks. While the most effective implementation may involve training transformers in a PICO-based way from scratch, we also present a cost-effective fine-tuning approach.

PICO: Secure Transformers via Robust Prompt Isolation and Cybersecurity Oversight

TL;DR

PICO addresses prompt injection in transformers by enforcing a principled invariant through dual-input channels: a frozen, immutable system-prompt stream and a trainable user-input stream. A gated fusion mechanism, enhanced by a Security Expert Agent and a Cybersecurity Knowledge Graph, dynamically biases toward the trusted system context to preserve safe, instruction-compliant outputs. The work provides formal guarantees on adversarial invariance and utility under benign inputs, and outlines concrete architectural realizations, training strategies, and case studies against policy-puppetry. While full training from scratch yields strongest security, the paper also presents an efficient fine-tuning pathway that leverages a pretrained model with dual streams and auxiliary security modules for practical deployment. Collectively, PICO integrates architectural isolation, multi-signal gating, and reinforcement learning to offer robust, end-to-end defenses with clear theoretical and practical benefits for secure LLM use in adversarial settings.

Abstract

We propose a robust transformer architecture designed to prevent prompt injection attacks and ensure secure, reliable response generation. Our PICO (Prompt Isolation and Cybersecurity Oversight) framework structurally separates trusted system instructions from untrusted user inputs through dual channels that are processed independently and merged only by a controlled, gated fusion mechanism. In addition, we integrate a specialized Security Expert Agent within a Mixture-of-Experts (MoE) framework and incorporate a Cybersecurity Knowledge Graph (CKG) to supply domain-specific reasoning. Our training design further ensures that the system prompt branch remains immutable while the rest of the network learns to handle adversarial inputs safely. This PICO framework is presented via a general mathematical formulation, then elaborated in terms of the specifics of transformer architecture, and fleshed out via hypothetical case studies including Policy Puppetry attacks. While the most effective implementation may involve training transformers in a PICO-based way from scratch, we also present a cost-effective fine-tuning approach.
Paper Structure (36 sections, 3 theorems, 26 equations, 1 figure)

This paper contains 36 sections, 3 theorems, 26 equations, 1 figure.

Table of Contents

  1. Introduction and Motivation
  2. Mathematical Formalization of the Security Strategy
  3. Formal Guarantees for Secure Transformer Architecture
  4. Preliminaries and Definitions
  5. Invariance under Adversarial Perturbations
  6. Probabilistic Guarantees via Security Signals
  7. Utility Preservation for Benign Inputs
  8. Discussion of Conditions
  9. Architectural Realization of the Security Strategy
  10. Input Processing Architecture
  11. Dual Input Channels for System and User
  12. Isolated Position Encodings
  13. Enforcing No Cross-Contamination
  14. Immutable System Prompt Representations
  15. Gated Fusion Mechanism for Controlled Integration
  16. ...and 21 more sections

Key Result

Theorem 3.1

Fix $\varepsilon>0$. Suppose for every adversarially perturbed input $U'$, the effective gate satisfies Then for all such $U'$, Consequently, by Lipschitz continuity of $D$,

Figures (1)

  • Figure 1: Fine-Tuning Based version of PICO Dual-Stream Secure Transformer Architecture. The pretrained transformer base is used to generate shared representations. Its output is processed along two branches: the system prompt branch is frozen to preserve trusted instructions, while the user branch is fine-tuned (with additional adapter modules). Security modules, including a Security Expert Agent and a Cybersecurity Knowledge Graph, provide dynamic inputs to a gated fusion module that combines both streams. The fused representation is then fed into a modified decoder for secure output generation.

Theorems & Definitions (6)

  • Theorem 3.1: Adversarial Invariance
  • proof
  • Theorem 3.2: Probabilistic Detection
  • proof
  • Theorem 3.3: Benign Utility Bound
  • proof