Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Poisoning Robustness Certification for Natural Language Generation

Mihnea Ghitu, Matthew Wicker

TL;DR

The work tackles the lack of provable robustness for autoregressive natural language generation under training-time data poisoning by formalizing two security properties: stability (robustness to any generation change) and validity (robustness to targeted harmful outputs). It introduces Targeted Partition Aggregation (TPA), a novel algorithm that certifies validity by computing the minimum poisoning budget needed to induce a specific harmful token or phrase, and extends this with a MILP-based multi-turn certification to tighten guarantees across sequences and prompts. The framework covers token-level, sequential, phrase-level, and multi-turn granularities, and validates the approach with empirical results on agent tool-calling, preference-based alignment, and backdoor attacks, showing substantial certified radii and improved robustness (e.g., >90% validity certification at $k=9$ with long horizons) at the cost of inference latency. These contributions enable more secure deployment of language models in security-critical contexts, while recognizing that latency and other attack surfaces remain open challenges for future work.

Abstract

Understanding the reliability of natural language generation is critical for deploying foundation models in security-sensitive domains. While certified poisoning defenses provide provable robustness bounds for classification tasks, they are fundamentally ill-equipped for autoregressive generation: they cannot handle sequential predictions or the exponentially large output space of language models. To establish a framework for certified natural language generation, we formalize two security properties: stability (robustness to any change in generation) and validity (robustness to targeted, harmful changes in generation). We introduce Targeted Partition Aggregation (TPA), the first algorithm to certify validity/targeted attacks by computing the minimum poisoning budget needed to induce a specific harmful class, token, or phrase. Further, we extend TPA to provide tighter guarantees for multi-turn generations using mixed integer linear programming (MILP). Empirically, we demonstrate TPA's effectiveness across diverse settings including: certifying validity of agent tool-calling when adversaries modify up to 0.5% of the dataset and certifying 8-token stability horizons in preference-based alignment. Though inference-time latency remains an open challenge, our contributions enable certified deployment of language models in security-critical applications.

Towards Poisoning Robustness Certification for Natural Language Generation

TL;DR

The work tackles the lack of provable robustness for autoregressive natural language generation under training-time data poisoning by formalizing two security properties: stability (robustness to any generation change) and validity (robustness to targeted harmful outputs). It introduces Targeted Partition Aggregation (TPA), a novel algorithm that certifies validity by computing the minimum poisoning budget needed to induce a specific harmful token or phrase, and extends this with a MILP-based multi-turn certification to tighten guarantees across sequences and prompts. The framework covers token-level, sequential, phrase-level, and multi-turn granularities, and validates the approach with empirical results on agent tool-calling, preference-based alignment, and backdoor attacks, showing substantial certified radii and improved robustness (e.g., >90% validity certification at with long horizons) at the cost of inference latency. These contributions enable more secure deployment of language models in security-critical contexts, while recognizing that latency and other attack surfaces remain open challenges for future work.

Abstract

Understanding the reliability of natural language generation is critical for deploying foundation models in security-sensitive domains. While certified poisoning defenses provide provable robustness bounds for classification tasks, they are fundamentally ill-equipped for autoregressive generation: they cannot handle sequential predictions or the exponentially large output space of language models. To establish a framework for certified natural language generation, we formalize two security properties: stability (robustness to any change in generation) and validity (robustness to targeted, harmful changes in generation). We introduce Targeted Partition Aggregation (TPA), the first algorithm to certify validity/targeted attacks by computing the minimum poisoning budget needed to induce a specific harmful class, token, or phrase. Further, we extend TPA to provide tighter guarantees for multi-turn generations using mixed integer linear programming (MILP). Empirically, we demonstrate TPA's effectiveness across diverse settings including: certifying validity of agent tool-calling when adversaries modify up to 0.5% of the dataset and certifying 8-token stability horizons in preference-based alignment. Though inference-time latency remains an open challenge, our contributions enable certified deployment of language models in security-critical applications.
Paper Structure (25 sections, 7 theorems, 11 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 25 sections, 7 theorems, 11 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Threat Model
  5. Provably Secure Language Generation
  6. The Autoregressive Certification Challenge
  7. Token-Level Certificates
  8. Sequential Multi-Token Certification
  9. Phrase-Level Certification
  10. Multi-Turn Collective Certification
  11. Experimental Results
  12. Evaluation Metrics
  13. Validity Certification for Agent Tool-Calling
  14. Certification of Secure Alignment
  15. Attack Robustness
  16. ...and 10 more sections

Key Result

Theorem 4.3

The value $r_{i}^{t}$ computed by Algorithm alg:tpa is a sound lower bound on the $i^{\text{th}}$-token validity radius (Definition def:ith-token-valid). For any poisoning budget $k \leq r_{i}^{t}$, the adversary cannot make target token $t$ the plurality prediction.

Figures (3)

  • Figure 1: An example of an agent responding with formal poisoning robustness certificates which enables users or agents to make security-informed decisions. Blue highlights indicate stability certificates (minimum training points an adversary must corrupt to change any token). Green highlights indicate validity certificates (minimum training points needed to induce a specific targeted harmful generation).
  • Figure 2: TPA certification results. (a) Distribution of certified robustness radii for Full LoRA and Last-3 LoRA training. Dashed lines indicate medians. (b) Accuracy comparison showing Full LoRA achieves 67% vs. 54% for Last-3 LoRA and 40% for zero-shot. (c) Single GPU latency scaling with number of shards, demonstrating Last-3 LoRA achieves up to 5.9$\times$ speedup over Full LoRA; zero-shot inference takes 0.3s.
  • Figure 3: Phrase-level certified robustness as a function of robustness radius. Each curve represents the percentage of examples certified at different phrase indices ($m=5$ tokens per phrase). Top row: Certified stability for generations up to 25 tokens. Bottom row: Certified validity for generations up to 60 tokens. Robustness increases at higher phrase indices due to the collapse toward boilerplate or harmless responses. Left column: OLMo-1B. Right column: Gemma2-2B.

Theorems & Definitions (12)

  • Definition 4.1: $i^{\text{th}}$-token stability
  • Definition 4.2: $i^{\text{th}}$-token validity
  • Theorem 4.3
  • proof
  • Definition 4.4: Finite-horizon validity & stability
  • Proposition 4.5: Sequential stability/validity certificate
  • Proposition 4.6: Phrase-level certificate
  • Definition 4.7: Collective TPA certificate
  • Theorem 4.8
  • Theorem 1.1: Collective/Multi-sample certificates for DPA
  • ...and 2 more