In-Context Bias Propagation in LLM-Based Tabular Data Generation

TL;DR

This work shows that in-context biases in prompts can systematically propagate to LLM-generated tabular data, distorting downstream fairness across univariate, conditional, and intersectional dimensions. The authors model the synthetic distribution as a two-component mixture and quantify drift with a drift score to reveal linear propagation patterns that intensify with larger context windows. They introduce adversarial in-context bias injection as a safety risk, demonstrating that a small fraction of biased demonstrations can degrade fairness in downstream classifiers while preserving utility. They also evaluate in-context preprocessing defenses and find partial mitigation, underscoring the need for stronger, more robust safeguards in LLM-based synthetic data pipelines.

Abstract

Large Language Models (LLMs) are increasingly used for synthetic tabular data generation through in-context learning (ICL), offering a practical solution for data augmentation in data scarce scenarios. While prior work has shown the potential of LLMs to improve downstream task performance through augmenting underrepresented groups, these benefits often assume access to a subset of unbiased in-context examples, representative of the real dataset. In real-world settings, however, data is frequently noisy and demographically skewed. In this paper, we systematically study how statistical biases within in-context examples propagate to the distribution of synthetic tabular data, showing that even mild in-context biases lead to global statistical distortions. We further introduce an adversarial scenario where a malicious contributor can inject bias into the synthetic dataset via a subset of in-context examples, ultimately compromising the fairness of downstream classifiers for a targeted and protected subgroup. Finally, we evaluate mitigation strategies based on preprocessing in-context examples, demonstrating that while such interventions can attenuate disparity, the inherent sensitivity of LLMs to adversarial prompts remains a persistent challenge. Our findings highlight a critical new vulnerability in LLM-based data generation pipelines within sensitive domains.

Figures (19)

• Figure 1: End-to-end bias propagation. An adversarial user injects biased examples into the prompt for a targeted group. The LLM replicates these statistical patterns via in-context learning, propagating the bias into the synthetic tabular data, and further compromising downstream fairness.
• Figure 2: Marginal in-context bias propagation (Mixtral-8x7b, $k=80$). (Left) Drift in the generated distribution $D_f(\mathcal{D}_G, \mathcal{D}_0)$ versus drift in the prompt distribution $D_f(\mathcal{D}_P, \mathcal{D}_0)$. The linear relationship shows that distributional shifts in prompts are proportionally transferred to synthetic outputs. (Right) Probability of the target group (Race/Gender) in generated samples increases linearly with prompt bias intensity $\pi$, confirming that LLMs propagate univariate demographic skews from in-context examples.
• Figure 3: Marginal bias propagation across model families. (Left) Probability of the target group in generated samples increases linearly with prompt bias intensity $\pi$ for all models at $k=80$. The dashed line shows the anchor baseline $p_{\mathcal{D}_0}(\text{Target})$. (Right) Propagation coefficient $\beta_k$ increases monotonically with context size $k$ across all models.
• Figure 4: Bias propagation strength increases with context size. Each subfigure shows results for a different context size $k \in \{20, 40, 60, 80\}$. The red curve shows the target group probability in generated samples $p_{\mathcal{D}_G}(\text{Target})$, the blue curve shows the target group probability in prompt examples $p_{\mathcal{D}_P}(\text{Target})$, and the dashed line shows anchor baseline $p_{\mathcal{D}_0}(\text{Target})$. As $k$ increases, the red curve converges toward the blue curve, indicating that larger context windows lead to stronger replication of prompt-level demographic distributions.
• Figure 5: Conditional bias propagation for targeted and non-targeted subgroups. (Left) Conditional probability of generating High Income examples for each gender subgroup as the bias intensity $\pi$ increases. The targeted subgroup exhibits strong linear sensitivity to $\pi$, while the non-targeted subgroup remains near 0.5. (Right) Statistical parity difference (SPD) between subgroups increases more steeply at $k=80$ than $k=20$, demonstrating that larger context windows amplify conditional bias propagation.