Automated Profile Inference with Language Model Agents

TL;DR

The paper identifies automated profile inference as a new privacy threat enabled by LLM-based agents that autonomously extract sensitive attributes from publicly available pseudonymous activity. It introduces AutoProfiler, a four-agent, multi-step framework that scrapes, analyzes, and infers personal information, achieving high accuracy and substantial efficiency gains over manual profiling across real and synthetic datasets. Experimental results on Reddit, Twitter, and SynthPAI demonstrate risks including de-anonymization and SPI leakage, while also evaluating robustness, calibration, and cross-LLM performance. The authors discuss mitigation strategies across user education, platform controls, model alignment, and privacy-enhancing technologies, and advocate for heightened awareness and policy consideration. Collectively, the work highlights a practical, scalable privacy threat with broad implications for online pseudonymity and privacy protection.

Abstract

Impressive progress has been made in automated problem-solving by the collaboration of large language models (LLMs) based agents. However, these automated capabilities also open avenues for malicious applications. In this paper, we study a new threat that LLMs pose to online pseudonymity, called automated profile inference, where an adversary can instruct LLMs to automatically scrape and extract sensitive personal attributes from publicly visible user activities on pseudonymous platforms. We also introduce an automated profiling framework called AutoProfiler to assess the feasibility of such threats in real-world scenarios. AutoProfiler consists of four specialized LLM agents, who work collaboratively to collect and process user online activities and generate a profile with extracted personal information. Experimental results on two real-world datasets and one synthetic dataset demonstrate that AutoProfiler is highly effective and efficient, and can be easily deployed on a web scale. We demonstrate that the inferred attributes are both sensitive and identifiable, posing significant risks of privacy breaches, such as de-anonymization and sensitive information leakage. Additionally, we explore mitigation strategies from different perspectives and advocate for increased public awareness of this emerging privacy threat to online pseudonymity.

Figures (15)

• Figure 1: Illustration of automated profile inference. An adversary instructs $\mathsf{AutoProfiler}$ to autonomously scrapeand analyze target users' online activities, extract personal attributes from these activities, and generate detailed user profiles that may cause privacy breaches.
• Figure 2: Illustration of the key profiling processes in $\mathsf{AutoProfiler}$. Upper left: It employs four specialized agents to complete the task (full prompts are provided in Appendix \ref{['Appendix:prompts_agents']}). Bottom left: Strategist coordinates other agents to sequentially retrieve, infer, and refine personal attributes. Right: Structured output of agents for efficient communication. Best viewed in color.
• Figure 3: Demonstration of inferred attributes on Reddit dataset, the sensitive information is masked with "***". $\mathsf{AutoProfiler}$ captures subtle clues (e.g., height) that users inadvertently reveal in seemingly insensitive contexts (e.g., car selection). We use GPT-4 to summarize these attributes for readability. Best viewed in color.
• Figure 4: Analysis of the categorized attributes of Reddit users by category, count, and estimated privacy risks.
• Figure 5: Inferring personal attributes from anonymized Tweets. "Invisible string" and "Dahlia" are masked as "***" as they refer to a song title and a person’s name. $\mathsf{AutoProfiler}$ still uncovers personal information like cultural background and family situation through subtle clues.

Theorems & Definitions (2)

• Definition 1: $(n, k)$-Deanonymization
• Definition 2: Calibration Accuracy