Log-linear Guardedness and its Implications
Shauli Ravfogel, Yoav Goldberg, Ryan Cotterell
TL;DR
The paper formalizes concept erasure through the lens of log-linear guardedness, using $\mathcal{V}$-information to quantify how much protected-attribute information remains after applying a guarding function $h$ to representations. It demonstrates that, with a binary downstream task and a discretized log-linear model family $\mathcal{V}^{\delta}$, leaked information is tightly bounded ($I_{\mathcal{V}^{\delta}}(\widehat{\mathrm{Y}} \to \mathrm{Z}) < \varepsilon$). However, for multiclass downstream tasks with a softmax, guarded representations can still leak substantial information about the protected attribute via appropriately structured $K$-Voronoi distributions, challenging the completeness of linear erasure as a bias mitigation. The experiments using RLACE to guard BERT representations on the Bias in Bios dataset corroborate the theory: binary leakage is reduced but not eliminated, while multiclass settings can reveal the protected attribute, highlighting the need for caution in applying linear erasure methods and motivating further study of intrinsic vs extrinsic bias. Overall, the work clarifies the limitations of log-linear guardedness and motivates new approaches that consider the specifics of downstream classifiers and task structure.
Abstract
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. However, the impact of this removal on the behavior of downstream classifiers trained on the modified representations is not fully understood. In this work, we formally define the notion of log-linear guardedness as the inability of an adversary to predict the concept directly from the representation, and study its implications. We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept. However, we demonstrate that a multiclass log-linear model \emph{can} be constructed that indirectly recovers the concept in some cases, pointing to the inherent limitations of log-linear guardedness as a downstream bias mitigation technique. These findings shed light on the theoretical limitations of linear erasure methods and highlight the need for further research on the connections between intrinsic and extrinsic bias in neural models.