BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel

TL;DR

The paper identifies a generation-budget side channel in Retrieval-Augmented Generation (RAG) systems as a practical vector for membership inference attacks. It introduces BudgetLeak, with BudgetLeak-P (partial-knowledge) and BudgetLeak-Z (zero-knowledge) variants, which exploit how outputs scale with the generation budget to distinguish member from non-member data. Extensive experiments across four datasets, three LLMs, and two retrievers show BudgetLeak achieving superior accuracy, AUC, and TPR at very low FPR compared to baselines, and demonstrate robustness to defenses like query or response rewriting. The work highlights a new privacy risk in RAG deployments and motivates defenses to protect external knowledge stores and sensitive corpora.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models by integrating external knowledge, but reliance on proprietary or sensitive corpora poses various data risks, including privacy leakage and unauthorized data usage. Membership inference attacks (MIAs) are a common technique to assess such risks, yet existing approaches underperform in RAG due to black-box constraints and the absence of strong membership signals. In this paper, we identify a previously unexplored side channel in RAG systems: the generation budget, which controls the maximum number of tokens allowed in a generated response. Varying this budget reveals observable behavioral patterns between member and non-member queries, as members gain quality more rapidly with larger budgets. Building on this insight, we propose BudgetLeak, a novel membership inference attack that probes responses under different budgets and analyzes metric evolution via sequence modeling or clustering. Extensive experiments across four datasets, three LLM generators, and two retrievers demonstrate that BudgetLeak consistently outperforms existing baselines, while maintaining high efficiency and practical viability. Our findings reveal a previously overlooked data risk in RAG systems and highlight the need for new defenses.

Figures (12)

• Figure 1: Distribution of similarity scores and response accuracy on RAG with LLaMA over HealthCareMagic-100k.
• Figure 2: Metric change rate and cumulative fluctuation distribution under varying generation budgets in RAG with LLaMA on HealthCareMagic-100k.
• Figure 3: Differences in cross-metric correlations between members and non-members in RAG with LLaMA on HealthCareMagic-100k.
• Figure 4: Overview of BudgetLeak.
• Figure 5: Performance of various attacks against RAGs on the HealthCareMagic-100k dataset (BudgetLeak-P).