Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage

Hanyin Shao, Jie Huang, Shen Zheng, Kevin Chen-Chuan Chang

TL;DR

This work quantifies how large language models (LLMs) form associations between pieces of information and the privacy implications of such associations. Using GPT-Neo/J/NeoX models across the LAMA (commonsense/factual knowledge) and Enron Email datasets, it defines an Association Easiness Score to capture how proximity and frequency of co-occurrence in training data affect inferential ability. The study finds that larger models exhibit stronger association, with higher accuracy for more frequently co-occurring pairs and shorter distances, though a clear performance gap exists between commonsense and PII tasks. It also shows that, while PII leakage via association is generally smaller than memorization, non-negligible risks remain—e.g., a 20B model can correctly predict about 3% of emails and 1% of phone numbers with targeted prompts—highlighting the need for vigilant mitigation and responsible deployment of powerful LLMs.

Abstract

The advancement of large language models (LLMs) brings notable improvements across various applications, while simultaneously raising concerns about potential private data exposure. One notable capability of LLMs is their ability to form associations between different pieces of information, but this raises concerns when it comes to personally identifiable information (PII). This paper delves into the association capabilities of language models, aiming to uncover the factors that influence their proficiency in associating information. Our study reveals that as models scale up, their capacity to associate entities/information intensifies, particularly when target pairs demonstrate shorter co-occurrence distances or higher co-occurrence frequencies. However, there is a distinct performance gap when associating commonsense knowledge versus PII, with the latter showing lower accuracy. Despite the proportion of accurately predicted PII being relatively small, LLMs still demonstrate the capability to predict specific instances of email addresses and phone numbers when provided with appropriate prompts. These findings underscore the potential risk to PII confidentiality posed by the evolving capabilities of LLMs, especially as they continue to expand in scale and power.

Quantifying Association Capabilities of Large Language Models and Its Implications on Privacy Leakage

TL;DR

This work quantifies how large language models (LLMs) form associations between pieces of information and the privacy implications of such associations. Using GPT-Neo/J/NeoX models across the LAMA (commonsense/factual knowledge) and Enron Email datasets, it defines an Association Easiness Score to capture how proximity and frequency of co-occurrence in training data affect inferential ability. The study finds that larger models exhibit stronger association, with higher accuracy for more frequently co-occurring pairs and shorter distances, though a clear performance gap exists between commonsense and PII tasks. It also shows that, while PII leakage via association is generally smaller than memorization, non-negligible risks remain—e.g., a 20B model can correctly predict about 3% of emails and 1% of phone numbers with targeted prompts—highlighting the need for vigilant mitigation and responsible deployment of powerful LLMs.

Abstract

The advancement of large language models (LLMs) brings notable improvements across various applications, while simultaneously raising concerns about potential private data exposure. One notable capability of LLMs is their ability to form associations between different pieces of information, but this raises concerns when it comes to personally identifiable information (PII). This paper delves into the association capabilities of language models, aiming to uncover the factors that influence their proficiency in associating information. Our study reveals that as models scale up, their capacity to associate entities/information intensifies, particularly when target pairs demonstrate shorter co-occurrence distances or higher co-occurrence frequencies. However, there is a distinct performance gap when associating commonsense knowledge versus PII, with the latter showing lower accuracy. Despite the proportion of accurately predicted PII being relatively small, LLMs still demonstrate the capability to predict specific instances of email addresses and phone numbers when provided with appropriate prompts. These findings underscore the potential risk to PII confidentiality posed by the evolving capabilities of LLMs, especially as they continue to expand in scale and power.
Paper Structure (20 sections, 2 equations, 6 figures, 2 tables)

This paper contains 20 sections, 2 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background and Problem Formulation
  4. Model and Data
  5. GPT-Neo, GPT-J, GPT-NeoX, and the Pile
  6. LAnguage Model Analysis Dataset
  7. Enron Email Dataset
  8. Method
  9. Prompt Construction
  10. Assessment of Association Easiness
  11. Evaluation of Model Prediction
  12. Overview of Results
  13. Analysis: Association Capability
  14. Common Factors Affecting Language Model Association
  15. Disparity in Association Performance
  16. ...and 5 more sections

Figures (6)

  • Figure 1: Testing procedure. The designed prompts are fed into the models. The output text is compared to the ground truth to determine if the prediction is correct.
  • Figure 2: LAMA Prediction Accuracy vs. Co-occurrence Distance.
  • Figure 3: Enron Email Prediction Accuracy vs. Co-occurrence Distance.
  • Figure 4: Prediction Accuracy vs. Co-occurrence Frequency.
  • Figure 5: Prediction Accuracy vs. Association Easiness Score.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 1
  • Definition 2