Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation

TL;DR

The paper investigates both unintended and malicious disclosure risks in code-generation LLMs trained on large, mined software repositories. It extends memorization risk analysis to include unintentional disclosure and demonstrates the approach on OLMo models with the Dolma dataset, using data-mining to assess sensitive information such as emails, phone numbers, and API keys. Findings show that dataset composition and processing changes can shift risk in complex ways, sometimes reducing one risk while increasing another, and that disclosure rates remain low but non-negligible at scale. The work emphasizes independent privacy/security assessments and responsible disclosure as essential practices for the LLM training data supply chain, with practical implications for pre-release testing and data sanitization across releases.

Abstract

This paper explores the risk that a large language model (LLM) trained for code generation on data mined from software repositories will generate content that discloses sensitive information included in its training data. We decompose this risk, known in the literature as ``unintended memorization,'' into two components: unintentional disclosure (where an LLM presents secrets to users without the user seeking them out) and malicious disclosure (where an LLM presents secrets to an attacker equipped with partial knowledge of the training data). We observe that while existing work mostly anticipates malicious disclosure, unintentional disclosure is also a concern. We describe methods to assess unintentional and malicious disclosure risks side-by-side across different releases of training datasets and models. We demonstrate these methods through an independent assessment of the Open Language Model (OLMo) family of models and its Dolma training datasets. Our results show, first, that changes in data source and processing are associated with substantial changes in unintended memorization risk; second, that the same set of operational changes may increase one risk while mitigating another; and, third, that the risk of disclosing sensitive information varies not only by prompt strategies or test datasets but also by the types of sensitive information. These contributions rely on data mining to enable greater privacy and security testing required for the LLM training data supply chain.

Figures (4)

• Figure 1: Count of potentially sensitive information in the Dolma training data. Results show that the count of email addresses and secret keys generally decreased between dataset releases, but the number of phone numbers increased. Analysis conducted on 89,937,427 code snippets from Dolma-v1.5s and 93,856,361 code snippets from Dolma-v1.7, aggregated across seven programming languages.
• Figure 2: Model propensity to disclose sensitive information in training data. Results show that the likelihood of unintentional disclosure decreased over iterated releases of OLMo-7B, while the likelihood of malicious disclosure increased. Analysis conducted on 9,711 unique prompts for malicious disclosure and 7,973 unique prompts for unintentional disclosure for both OLMo models.
• Figure 3: Malicious disclosure of sensitive information across various prompting strategies based on pass@10 results. Results show that malicious disclosures increased for phone numbers and secret keys between releases of the OLMo-7B models. For each strategy, analysis conducted on 3,917 prompts prepared from Dolma-v1.5s and 2,179 prompts prepared from Dolma-v1.7 datasets for corresponding OLMo models.
• Figure 4: Unintentional disclosure of sensitive information across various test datasets based on pass@10 results. Results show that unintentional disclosure is rare for each type of sensitive information. Analysis conducted on a total of 2,897 unique prompts for both OLMo models, aggregated across test datasets.