Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation
Francisco Vargas, Ryan Cotterell
TL;DR
The paper addresses gender bias in pre-trained word representations by testing Bolukbasi et al.'s linear bias subspace through a kernelized nonlinear extension. It develops a kernel PCA based bias isolation in a reproducing kernel Hilbert space, with neutralize and equalize operations adapted to the feature space and strategies to compute pre-images back to input space. Across WEAT, professions, indirect bias, and SimLex-999 benchmarks, the nonlinear method yields results statistically indistinguishable from the linear approach, thereby supporting the linear subspace hypothesis. The findings suggest that a linear subspace is a sufficient and robust representation for gender bias in word embeddings, implying that simpler debiasing pipelines may suffice in practice, while the methodology provides a principled nonlinear generalization for future work.
Abstract
Bolukbasi et al. (2016) presents one of the first gender bias mitigation techniques for word representations. Their method takes pre-trained word representations as input and attempts to isolate a linear subspace that captures most of the gender bias in the representations. As judged by an analogical evaluation task, their method virtually eliminates gender bias in the representations. However, an implicit and untested assumption of their method is that the bias subspace is actually linear. In this work, we generalize their method to a kernelized, nonlinear version. We take inspiration from kernel principal component analysis and derive a nonlinear bias isolation technique. We discuss and overcome some of the practical drawbacks of our method for non-linear gender bias mitigation in word representations and analyze empirically whether the bias subspace is actually linear. Our analysis shows that gender bias is in fact well captured by a linear subspace, justifying the assumption of Bolukbasi et al. (2016).