Behind the Screens: Uncovering Bias in AI-Driven Video Interview Assessments Using Counterfactuals

TL;DR

This work targets bias in AI-driven video interview assessments used for hiring by introducing a counterfactual fairness framework based on multimodal video data. It develops a GAN-based counterfactual video generation pipeline that edits protected attributes in latent space while preserving other content, enabling fairness auditing without access to model internals. The authors formalize counterfactual fairness with the condition $P(\\hat{Y}_{A \\leftarrow a}(U) = y \\mid X = x, A = a) = P(\\hat{Y}_{A \\leftarrow a'}(U) = y \\mid X = x, A = a')$ and apply it to a state-of-the-art OCEAN-based personality predictor (MTDNN), supported by a protected-attribute classifier for validation. Across three experiments on the CLFI dataset, counterfactual edits reveal and mitigate demographic disparities in predicted traits and interview scores, demonstrating the framework’s power for fairness auditing in black-box video-based affective computing. The results advocate for broader application to audio and text modalities, more diverse datasets, and careful ethical governance to foster transparency in AI hiring technologies.

Abstract

AI-enhanced personality assessments are increasingly shaping hiring decisions, using affective computing to predict traits from the Big Five (OCEAN) model. However, integrating AI into these assessments raises ethical concerns, especially around bias amplification rooted in training data. These biases can lead to discriminatory outcomes based on protected attributes like gender, ethnicity, and age. To address this, we introduce a counterfactual-based framework to systematically evaluate and quantify bias in AI-driven personality assessments. Our approach employs generative adversarial networks (GANs) to generate counterfactual representations of job applicants by altering protected attributes, enabling fairness analysis without access to the underlying model. Unlike traditional bias assessments that focus on unimodal or static data, our method supports multimodal evaluation-spanning visual, audio, and textual features. This comprehensive approach is particularly important in high-stakes applications like hiring, where third-party vendors often provide AI systems as black boxes. Applied to a state-of-the-art personality prediction model, our method reveals significant disparities across demographic groups. We also validate our framework using a protected attribute classifier to confirm the effectiveness of our counterfactual generation. This work provides a scalable tool for fairness auditing of commercial AI hiring platforms, especially in black-box settings where training data and model internals are inaccessible. Our results highlight the importance of counterfactual approaches in improving ethical transparency in affective computing.

Figures (8)

• Figure 1: Overview of the proposed framework utilizing counterfactual videos for bias detection in affective computing models.
• Figure 2: Architecture of the multi-task deep neural network model (MTDNN model) mujtaba2021multi used for personality and interview score prediction.
• Figure 3: Distribution of OCEANI dimension scores in the original dataset ground-truth labels, grouped by gender, ethnicity, and age. Each plot shows the distribution for all individuals, the protected group ("M" for male, "Af.Am." for African-American individuals, and "$\text{A} \ge 40$" for individuals aged 40 and above), and the unprotected group ("F" for female, "Asi.+Cau." for Asian and Caucasian individuals, and "$\text{A} < 40$" for individuals under 40).
• Figure 4: Distributions of predicted OCEANI dimension scores for individuals and their counterfactual counterparts. The first row shows predictions for male individuals using the full model and the changes in scores when their counterfactual videos are input. The second row shows predictions using the visual-only model (MTDNN-Visual-Inv) for the same individuals and their counterfactuals.
• Figure 5: Example video frames showing the original, GAN-inverted, and counterfactually modified images for three individuals. For each row, the first image is the original frame, the second is the GAN-inverted reconstruction, and the third is the counterfactual frame generated by manipulating a protected attribute via latent space boundary editing (first row: gender changed from male to female; second row: ethnicity changed to Caucasian; third row: age reduced for a male individual).

Theorems & Definitions (4)

• Definition 1.1: Counterfactual Fairness
• Definition 2.1: Fairness Through Unawareness
• Definition 2.2: Independence barocas2017fairness
• Definition 2.3: Disparate Impact mujtaba2019ethicalbarocas2017fairness