CanaryBench: Stress Testing Privacy Leakage in Cluster-Level Conversation Summaries

TL;DR

CanaryBench provides a practical stress test for privacy leakage in cluster-level summaries derived from conversations. It plants known canaries in synthetic data and measures verbatim leakage under extractive and non-extractive summarizers using LR_canary and LR_cluster metrics. The key finding is that extractive summaries leak canaries at high rates (cluster-level leakage around 0.96), while a simple defense combining a k_min publication threshold and regex-based redaction eliminates leakage with minimal loss of cluster coherence. The work offers actionable guidelines for production analytics and underscores the need for non-extractive summaries and robust privacy controls in published artifacts.

Abstract

Aggregate analytics over conversational data are increasingly used for safety monitoring, governance, and product analysis in large language model systems. A common practice is to embed conversations, cluster them, and publish short textual summaries describing each cluster. While raw conversations may never be exposed, these derived summaries can still pose privacy risks if they contain personally identifying information (PII) or uniquely traceable strings copied from individual conversations. We introduce CanaryBench, a simple and reproducible stress test for privacy leakage in cluster-level conversation summaries. CanaryBench generates synthetic conversations with planted secret strings ("canaries") that simulate sensitive identifiers. Because canaries are known a priori, any appearance of these strings in published summaries constitutes a measurable leak. Using TF-IDF embeddings and k-means clustering on 3,000 synthetic conversations (24 topics) with a canary injection rate of 0.60, we evaluate an intentionally extractive example snippet summarizer that models quote-like reporting. In this configuration, we observe canary leakage in 50 of 52 canary-containing clusters (cluster-level leakage rate 0.961538), along with nonzero regex-based PII indicator counts. A minimal defense combining a minimum cluster-size publication threshold (k-min = 25) and regex-based redaction eliminates measured canary leakage and PII indicator hits in the reported run while maintaining a similar cluster-coherence proxy. We position this work as a societal impacts contribution centered on privacy risk measurement for published analytics artifacts rather than raw user data.

Figures (5)

• Figure 1: CanaryBench evaluation pipeline (vertical flow). Synthetic conversations containing planted canaries are embedded using TF-IDF, clustered via k-means, and summarized. Published summaries can optionally be filtered (k-min threshold) and redacted (regex-based PII removal) before evaluation for canary leakage and PII indicators.
• Figure 2: Extractive summarization directly quotes conversation text, creating a path for canaries to appear in published summaries. Non-extractive approaches aggregate topics without verbatim reproduction, preventing direct leakage.
• Figure 3: Leakage rates under the extractive example-based summarizer. The defended configuration reduces both per-canary and cluster-level leakage to zero in the reported runs. Zero values are annotated with highlighted boxes for clarity.
• Figure 4: Regex-based PII indicator hits in published summaries. Counts drop to zero after applying redaction. Zero values are clearly annotated.
• Figure 5: Utility proxy (average cluster coherence) in the reported runs. Both configurations maintain similar coherence scores, suggesting privacy defenses do not substantially degrade topical quality.