Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models

Jigang Fan, Zhenghong Zhou, Ruofan Jin, Le Cong, Mengdi Wang, Zaixi Zhang

TL;DR

SafeProtein introduces a systematic red-teaming framework and a dedicated benchmark (SafeProtein-Bench) to assess biosafety risks in protein foundation models. It combines multimodal prompt engineering with heuristic beam search on diffusion-based generation to craft adversarial inputs, and evaluates them via joint sequence-structure criteria. Empirical results show substantial jailbreak susceptibility in state-of-the-art models (up to 70% on ESM3) and demonstrate that richer structure prompts and advanced generation strategies can amplify risks, including design-capability observations. The work underscores significant biosafety concerns and provides a practical platform and guidelines for developing safer frontier protein models and governance frameworks.

Abstract

Proteins play crucial roles in almost all biological processes. The advancement of deep learning has greatly accelerated the development of protein foundation models, leading to significant successes in protein understanding and design. However, the lack of systematic red-teaming for these models has raised serious concerns about their potential misuse, such as generating proteins with biological safety risks. This paper introduces SafeProtein, the first red-teaming framework designed for protein foundation models to the best of our knowledge. SafeProtein combines multimodal prompt engineering and heuristic beam search to systematically design red-teaming methods and conduct tests on protein foundation models. We also curated SafeProtein-Bench, which includes a manually constructed red-teaming benchmark dataset and a comprehensive evaluation protocol. SafeProtein achieved continuous jailbreaks on state-of-the-art protein foundation models (up to 70% attack success rate for ESM3), revealing potential biological safety risks in current protein foundation models and providing insights for the development of robust security protection technologies for frontier models. The codes will be made publicly available at https://github.com/jigang-fan/SafeProtein.

SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Models

TL;DR

SafeProtein introduces a systematic red-teaming framework and a dedicated benchmark (SafeProtein-Bench) to assess biosafety risks in protein foundation models. It combines multimodal prompt engineering with heuristic beam search on diffusion-based generation to craft adversarial inputs, and evaluates them via joint sequence-structure criteria. Empirical results show substantial jailbreak susceptibility in state-of-the-art models (up to 70% on ESM3) and demonstrate that richer structure prompts and advanced generation strategies can amplify risks, including design-capability observations. The work underscores significant biosafety concerns and provides a practical platform and guidelines for developing safer frontier protein models and governance frameworks.

Abstract

Proteins play crucial roles in almost all biological processes. The advancement of deep learning has greatly accelerated the development of protein foundation models, leading to significant successes in protein understanding and design. However, the lack of systematic red-teaming for these models has raised serious concerns about their potential misuse, such as generating proteins with biological safety risks. This paper introduces SafeProtein, the first red-teaming framework designed for protein foundation models to the best of our knowledge. SafeProtein combines multimodal prompt engineering and heuristic beam search to systematically design red-teaming methods and conduct tests on protein foundation models. We also curated SafeProtein-Bench, which includes a manually constructed red-teaming benchmark dataset and a comprehensive evaluation protocol. SafeProtein achieved continuous jailbreaks on state-of-the-art protein foundation models (up to 70% attack success rate for ESM3), revealing potential biological safety risks in current protein foundation models and providing insights for the development of robust security protection technologies for frontier models. The codes will be made publicly available at https://github.com/jigang-fan/SafeProtein.

Paper Structure

This paper contains 25 sections, 14 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. SafeProtein: Red-Teaming Methodology for Protein Models
  3. Problem Formulation.
  4. Input Prompts and Generation Strategies for Protein Red-Teaming
  5. Implementation Details of Generation Strategies
  6. SafeProtein-Bench: Benchmark for Red-Teaming Protein Models
  7. Benchmark Dataset Construction
  8. Evaluation Protocol
  9. Experiments
  10. Settings
  11. Input sequence mask construction.
  12. Implementation details.
  13. Evaluation metrics.
  14. Red-Teaming Results for ESM3 and DPLM2
  15. Results of Additional Generation Strategies for ESM3
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Overview of (A) SafeProtein and (B) SafeProtein-Bench.
  • Figure 2: Red-teaming results of (A) ESM3 and (B) DPLM2.
  • Figure 3: Red-teaming results of ESM3 on additional generation strategies.
  • Figure 4: Sequence similarity and RMSD distributions of (A) overall and (B) different generation strategies. Comparison of the predicted structures for (C) UniProt ID: P11407 and (D) UniProt ID: Q6STF1 with their native structures, based on the sequences generated by ESM3.
  • Figure S1: Details of SafeProtein-Bench dataset. (A) Taxonomic distribution of the test cases in our dataset. (B) Length distribution of the sequences in dataset.