Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT

Abhisek Dash, Soumi Das, Elisabeth Kirsten, Qinyuan Wu, Sai Keerthana Karnam, Krishna P. Gummadi, Thorsten Holz, Muhammad Bilal Zafar, Savvas Zannettou

TL;DR

The paper treats memory in ChatGPT as an Algorithmic Self-portrait, encoding users' self-disclosures to support personalized dialogue while raising questions about user agency and data privacy. Analyzing 2,050 memories from 80 GDPR-donated traces reveals that AI-initiated memory updates dominate (96%), with substantial GDPR personal data (28%) and psychological content (52%), yet most memories are grounded in the user context (84%). Provenance analyses show memories are largely faithful to user conversations, even as memory extraction can reveal more sensitive data, motivating the Attribution Shield framework. Attribution Shield demonstrates that privacy-preserving reformulations can reduce attribution while preserving task utility, offering a path toward safer personalization in conversational AI.

Abstract

To enable personalized and context-aware interactions, conversational AI systems have introduced a new mechanism: Memory. Memory creates what we refer to as the Algorithmic Self-portrait - a new form of personalization derived from users' self-disclosed information divulged within private conversations. While memory enables more coherent exchanges, the underlying processes of memory creation remain opaque, raising critical questions about data sensitivity, user agency, and the fidelity of the resulting portrait. To bridge this research gap, we analyze 2,050 memory entries from 80 real-world ChatGPT users. Our analyses reveal three key findings: (1) A striking 96% of memories in our dataset are created unilaterally by the conversational system, potentially shifting agency away from the user; (2) Memories, in our dataset, contain a rich mix of GDPR-defined personal data (in 28% memories) along with psychological insights about participants (in 52% memories); and (3)~A significant majority of the memories (84%) are directly grounded in user context, indicating faithful representation of the conversations. Finally, we introduce a framework-Attribution Shield-that anticipates these inferences, alerts about potentially sensitive memory inferences, and suggests query reformulations to protect personal information without sacrificing utility.

The Algorithmic Self-Portrait: Deconstructing Memory in ChatGPT

TL;DR

The paper treats memory in ChatGPT as an Algorithmic Self-portrait, encoding users' self-disclosures to support personalized dialogue while raising questions about user agency and data privacy. Analyzing 2,050 memories from 80 GDPR-donated traces reveals that AI-initiated memory updates dominate (96%), with substantial GDPR personal data (28%) and psychological content (52%), yet most memories are grounded in the user context (84%). Provenance analyses show memories are largely faithful to user conversations, even as memory extraction can reveal more sensitive data, motivating the Attribution Shield framework. Attribution Shield demonstrates that privacy-preserving reformulations can reduce attribution while preserving task utility, offering a path toward safer personalization in conversational AI.

Abstract

To enable personalized and context-aware interactions, conversational AI systems have introduced a new mechanism: Memory. Memory creates what we refer to as the Algorithmic Self-portrait - a new form of personalization derived from users' self-disclosed information divulged within private conversations. While memory enables more coherent exchanges, the underlying processes of memory creation remain opaque, raising critical questions about data sensitivity, user agency, and the fidelity of the resulting portrait. To bridge this research gap, we analyze 2,050 memory entries from 80 real-world ChatGPT users. Our analyses reveal three key findings: (1) A striking 96% of memories in our dataset are created unilaterally by the conversational system, potentially shifting agency away from the user; (2) Memories, in our dataset, contain a rich mix of GDPR-defined personal data (in 28% memories) along with psychological insights about participants (in 52% memories); and (3)~A significant majority of the memories (84%) are directly grounded in user context, indicating faithful representation of the conversations. Finally, we introduce a framework-Attribution Shield-that anticipates these inferences, alerts about potentially sensitive memory inferences, and suggests query reformulations to protect personal information without sacrificing utility.
Paper Structure (21 sections, 7 figures, 6 tables)

This paper contains 21 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Dataset for Memory Analyses
  4. User Agency and Sensitivity of Memories
  5. Agency of Memory Update
  6. Remembering Sensitive Information
  7. Memories and Mental States
  8. Taxonomy for Psychological Memories
  9. Characterization of Psychological Memories
  10. Provenance of Memories
  11. Methodology
  12. Observations
  13. Reverse Engineering Memories
  14. Experimental Framework
  15. Imitating Memory Extractor
  16. ...and 6 more sections

Figures (7)

  • Figure 1: (a) Only $4.1\%$ of memories ($n=84$) are initiated by participants. (b) 25 participants requested at least one memory update. ChatGPT triggers memory updates when it finds the context to be useful for future conversations.
  • Figure 2: In contrast to OpenAI policies, ChatGPT memories store information which could be categorized as GDPR defined personal data or special category personal data.
  • Figure 3: Distribution of memory entries among the different ToM categories as per GPT-4o annotations.
  • Figure 4: (a--b) Syntactic, (c) Semantic, and (d) LLM evaluation based provenance measures. Majority of ChatGPT’s memories are directly or semantically grounded in user input.
  • Figure 5: (a) Information gain of all predicted memories over ChatGPT memories shows high gain, hinting at higher potential risks. (b) Responses from rephrased queries from ICL and FT are semantically similar to the original response.
  • ...and 2 more figures