Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs

TL;DR

This paper introduces a multi-stage pipeline for scalable data filtering and shows that it offers a tractable and effective method for minimizing biothreat proxy knowledge in LLMs, and establishes pretraining data curation as a promising layer of defense for open-weight AI systems.

Abstract

Open-weight AI systems offer unique benefits, including enhanced transparency, open research, and decentralized access. However, they are vulnerable to tampering attacks which can efficiently elicit harmful behaviors by modifying weights or activations. Currently, there is not yet a robust science of open-weight model risk management. Existing safety fine-tuning methods and other post-training techniques have struggled to make LLMs resistant to more than a few dozen steps of adversarial fine-tuning. In this paper, we investigate whether filtering text about dual-use topics from training data can prevent unwanted capabilities and serve as a more tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable data filtering and show that it offers a tractable and effective method for minimizing biothreat proxy knowledge in LLMs. We pretrain multiple 6.9B-parameter models from scratch and find that they exhibit substantial resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M tokens of biothreat-related text -- outperforming existing post-training baselines by over an order of magnitude -- with no observed degradation to unrelated capabilities. However, while filtered models lack internalized dangerous knowledge, we find that they can still leverage such information when it is provided in context (e.g., via search tool augmentation), demonstrating a need for a defense-in-depth approach. Overall, these findings help to establish pretraining data curation as a promising layer of defense for open-weight AI systems.

Figures (9)

• Figure 1: Training data filtering makes LLMs resistant to adversarial fine-tuning without sacrificing general performance. Models whose training data has been filtered to remove text related to dual-use biology topics (left) have unaffected general capabilities and (right) have low biothreat proxy capabilities and resist up to 10,000 steps and 300M tokens of adversarial fine-tuning. We further detail results in \ref{['sec:tamper_resistant']}.
• Figure 2: Our multi-stage data filtering pipeline: Our goal is to filter out data related to unwanted topics. We study biothreat-proxy knowledge as a representative example. All documents undergo initial "blocklist" filtering, where those without prohibited terms are retained without further review. Documents containing blocked terms (e.g., "pathogen(s)") are escalated to a fine-tuned text classifier that evaluates semantic content. The classifier assigns probability scores for unsafe content: documents scoring below the predetermined threshold are retained, while those exceeding it are excluded from the training corpus. In practice, the vast majority of documents are approved by the blocklist and thus do not require review by the classifier stage. We further detail our methodology in \ref{['sec:filtering']}.
• Figure 3: Data filtering (our technique) performs competitively with Circuit-Breaking (CB) techniques under black-box evals and attacks. We evaluate data filtering approaches against baselines on general knowledge (higher is better) and biothreat proxy knowledge (lower is better). Dotted lines indicate random chance. We report performance across repeated non-deterministic attacks with error bars. Data filtering and CB methods are comparable: both have similarly minor effects on general capabilities, CB methods perform slightly better on MCQA biothreat proxy evaluations, and filtering methods perform slightly better on cloze biothreat proxy evaluations. Data filtering is robust to the input-space attacks, especially in the cloze-prompt setting. These results demonstrate that pretraining data filtering is effective at significantly preventing biothreat proxy knowledge, including random-chance-level performance on cloze-style prompts.
• Figure 4: Filtering biothreat proxy content from training data makes LLMs resist adversarial tampering. (Left & middle) Our LLMs are tamper-resistant up to 10,000 steps of fine-tuning on 305M tokens of biothreat proxy scientific text. (Right) Our LLMs are resistant to latent-space attacks competitively with Circuit-Breaking (CB) methods: CB+LAT performs better on multiple-choice evals while filtering performs better on cloze evals.
• Figure 6: Pretraining data filtering cannot prevent in-context retrieval of unwanted information, but Circuit-Breaking can. However, no models resist an ensemble fine-tuning + in-context-retrieval attack. Baseline and filtered models alike can perform well on our "open-book" biothreat knowledge tests in which a passage containing the answer is given in context. Circuit-Breaking complements filtering by impairing the model's ability to retrieve biothreat-related information in context. No defenses, however, resist our ensemble attack.