Towards Provably Secure Generative AI: Reliable Consensus Sampling

TL;DR

The paper addresses the lack of provable security in generative AI by advancing from Consensus Sampling (CS) to Reliable Consensus Sampling (RCS), which uses a trace-based, abstention-free framework to bound risk under Byzantine adversaries. It establishes formal model-group safety properties and proves a tight upper bound on output risk, $q(U) \le R \cdot \mu(U) + negl(\\lambda)$, with $R$ polynomial in the security parameter. A quantum-inspired extension, F-RCS, introduces a feedback mechanism that dynamically downweights potentially unsafe models, further boosting robustness (accuracy in identifying unsafe models ~90%). Extensive experiments show RCS delivers significantly higher Safe Rate and lower Abstention Rate than CS while maintaining latency comparable to CS, and it remains robust under collusion. Collectively, these contributions push toward provably secure generative AI with practical utility and adjustable safety guarantees.

Abstract

Existing research on generative AI security is primarily driven by mutually reinforcing attack and defense methodologies grounded in empirical experience. This dynamic frequently gives rise to previously unknown attacks that can circumvent current detection and prevention. This necessitates the continual updating of security mechanisms. Constructing generative AI with provable security and theoretically controllable risk is therefore necessary. Consensus Sampling (CS) is a promising algorithm toward provably secure AI. It controls risk by leveraging overlap in model output probabilities. However, we find that CS relies on frequent abstention to avoid unsafe outputs, which reduces utility. Moreover, CS becomes highly vulnerable when unsafe models are maliciously manipulated. To address these issues, we propose a new primitive called Reliable Consensus Sampling (RCS), that traces acceptance probability to tolerate extreme adversarial behaviors, improving robustness. RCS also eliminates the need for abstention entirely. We further develop a feedback algorithm to continuously and dynamically enhance the safety of RCS. We provide theoretical guarantees that RCS maintains a controllable risk threshold. Extensive experiments show that RCS significantly improves robustness and utility while maintaining latency comparable to CS. We hope this work contributes to the development of provably secure generative AI.

Figures (14)

• Figure 1: Overview of security risks in generative model reasoning. These risks arise from the aggregation of risks across three core stages. From a theoretical perspective, such risks are unavoidable.
• Figure 2: Model probability distribution for consensus sampling under diverse adversarial environments.
• Figure 3: Evaluation results for the safe rate when $f < \lceil \frac{n}{2} \rceil$.
• Figure 4: Evaluation results for the latency when $f < \lceil \frac{n}{2} \rceil$.
• Figure 5: Evaluation results for the safe rate when Byzantine models collude.