Integrating Fairness and Model Pruning Through Bi-level Optimization

TL;DR

This work introduces a novel concept of fair model pruning, which involves developing a sparse model that adheres to fairness criteria and proposes a framework to jointly optimize the pruning mask and weight update processes with fairness constraints.

Abstract

Deep neural networks have achieved exceptional results across a range of applications. As the demand for efficient and sparse deep learning models escalates, the significance of model compression, particularly pruning, is increasingly recognized. Traditional pruning methods, however, can unintentionally intensify algorithmic biases, leading to unequal prediction outcomes in critical applications and raising concerns about the dilemma of pruning practices and social justice. To tackle this challenge, we introduce a novel concept of fair model pruning, which involves developing a sparse model that adheres to fairness criteria. In particular, we propose a framework to jointly optimize the pruning mask and weight update processes with fairness constraints. This framework is engineered to compress models that maintain performance while ensuring fairness in a unified process. To this end, we formulate the fair pruning problem as a novel constrained bi-level optimization task and derive efficient and effective solving strategies. We design experiments across various datasets and scenarios to validate our proposed method. Our empirical analysis contrasts our framework with several mainstream pruning strategies, emphasizing our method's superiority in maintaining model fairness, performance, and efficiency.

Figures (8)

• Figure 1: The accuracy and bias of different pruning methods and Prune & Fair patterns.
• Figure 2: Sketch of the proposed framework.
• Figure 3: Accuracy and fairness of the pruned ResNet10 on CelebA and LFW datasets at varying sparsity levels for two predictive tasks in these datasets. There are two prediction tasks in two datasets: predicting Bags_Under_Eyes and Blond_Hair in CelebA, and predicting Wavy_Hair and Big_Nose in LFW. Subfigs. (a, b, c, d) indicate the comparison of accuracy between the proposed method and the baseline methods. Subfigs. (e, d, g, h) indicate the comparison of Compressed Model Fairness between the proposed method and the baseline methods. This figure shows the proposed methods (BiFP_str and BiFP_uns) are the only ones that ensure fairness while having comparable or even better accuracy.
• Figure 4: Accuracy and fairness of the pruned MobileNetV2 on CelebA and LFW datasets at varying sparsity levels. Subfigs. (a, b, c, d) indicate the comparison of accuracy between the proposed method and the baseline methods. Subfigs. (e, f, g, h) indicate the comparison of Compressed Model Fairness between the proposed method and the baseline methods. This figure shows the proposed methods (BiFP_str and BiFP_uns) are the only ones that ensure fairness while having comparable or even better accuracy.
• Figure 5: Training iterations used to obtain fair models using a two-stage pruning strategy. The red shade indicates the training iterations for BiFP. The dashed line indicates the performance of BiFP after stopping training for easy comparison with others.

Theorems & Definitions (3)

• Definition 1: Performance Fairness
• Definition 2: Compressed Model Performance Degradation
• Definition 3: Performance Degradation Fairness