Generating Synthetic Fair Syntax-agnostic Data by Learning and Distilling Fair Representation
Md Fahim Sikder, Resmi Ramachandranpillai, Daniel de Leng, Fredrik Heintz
TL;DR
This work introduces a syntax-agnostic fair generative framework that learns a fair latent representation via a distance-covariance regularized VAE and then distills this representation into a smaller encoder to enable efficient generation of fair synthetic data. The distilled latent space, decoded by a shared decoder, yields high-fidelity data with improved fairness and downstream utility, demonstrated on tabular and image benchmarks. By combining fairness-oriented representation learning with latent-space distillation and a KL-based utility loss, the approach reduces computational overhead while outperforming state-of-the-art fair generative models across multiple metrics. The study also provides extensive analyses, including visual, explainability, and runtime evaluations, and discusses limitations and social impacts with a path toward multi-attribute fairness.
Abstract
Data Fairness is a crucial topic due to the recent wide usage of AI powered applications. Most of the real-world data is filled with human or machine biases and when those data are being used to train AI models, there is a chance that the model will reflect the bias in the training data. Existing bias-mitigating generative methods based on GANs, Diffusion models need in-processing fairness objectives and fail to consider computational overhead while choosing computationally-heavy architectures, which may lead to high computational demands, instability and poor optimization performance. To mitigate this issue, in this work, we present a fair data generation technique based on knowledge distillation, where we use a small architecture to distill the fair representation in the latent space. The idea of fair latent space distillation enables more flexible and stable training of Fair Generative Models (FGMs). We first learn a syntax-agnostic (for any data type) fair representation of the data, followed by distillation in the latent space into a smaller model. After distillation, we use the distilled fair latent space to generate high-fidelity fair synthetic data. While distilling, we employ quality loss (for fair distillation) and utility loss (for data utility) to ensure that the fairness and data utility characteristics remain in the distilled latent space. Our approaches show a 5%, 5% and 10% rise in performance in fairness, synthetic sample quality and data utility, respectively, than the state-of-the-art fair generative model.