Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches

Obed Junias, Prajakta Kini, Theodora Chaspari

TL;DR

This study tackles algorithmic bias in language-based depression detection by comparing DNN-based embeddings with few-shot LLM approaches on the DAIC-WOZ clinical interviews, focusing on gender and race/ethnicity. It combines fairness-aware loss functions for DNNs and in-context prompting with ethical framing for LLMs, evaluating performance and fairness using metrics such as $BA$ and $EO$ across demographic groups. Key findings show LLMs generally outperform DNNs and exhibit reduced gender bias, while racial disparities persist; among DNN debiasing methods, the worst-group loss offers a favorable balance between accuracy and fairness, whereas the fairness-regularized loss increases $EO$ at the cost of accuracy. The results underscore the need for demographically aware, model-specific debiasing strategies in clinical NLP and highlight limitations related to underrepresented groups and multilingual generalizability.

Abstract

This paper investigates algorithmic bias in language-based models for automated depression detection, focusing on socio-demographic disparities related to gender and race/ethnicity. Models trained using deep neural networks (DNN) based embeddings are compared to few-shot learning approaches with large language models (LLMs), evaluating both performance and fairness on clinical interview transcripts from the Distress Analysis Interview Corpus/Wizard-of-Oz (DAIC-WOZ). To mitigate bias, fairness-aware loss functions are applied to DNN-based models, while in-context learning with varied prompt framing and shot counts is explored for LLMs. Results indicate that LLMs outperform DNN-based models in depression classification, particularly for underrepresented groups such as Hispanic participants. LLMs also exhibit reduced gender bias compared to DNN-based embeddings, though racial disparities persist. Among fairness-aware techniques for mitigating bias in DNN-based embeddings, the worst-group loss, which is designed to minimize loss for the worst-performing demographic group, achieves a better balance between performance and fairness. In contrast, the fairness-regularized loss minimizes loss across all groups but performs less effectively. In LLMs, guided prompting with ethical framing helps mitigate gender bias in the 1-shot setting. However, increasing the number of shots does not lead to further reductions in disparities. For race/ethnicity, neither prompting strategy nor increasing $N$ in $N$-shot learning effectively reduces disparities.

Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches

TL;DR

This study tackles algorithmic bias in language-based depression detection by comparing DNN-based embeddings with few-shot LLM approaches on the DAIC-WOZ clinical interviews, focusing on gender and race/ethnicity. It combines fairness-aware loss functions for DNNs and in-context prompting with ethical framing for LLMs, evaluating performance and fairness using metrics such as and across demographic groups. Key findings show LLMs generally outperform DNNs and exhibit reduced gender bias, while racial disparities persist; among DNN debiasing methods, the worst-group loss offers a favorable balance between accuracy and fairness, whereas the fairness-regularized loss increases at the cost of accuracy. The results underscore the need for demographically aware, model-specific debiasing strategies in clinical NLP and highlight limitations related to underrepresented groups and multilingual generalizability.

Abstract

This paper investigates algorithmic bias in language-based models for automated depression detection, focusing on socio-demographic disparities related to gender and race/ethnicity. Models trained using deep neural networks (DNN) based embeddings are compared to few-shot learning approaches with large language models (LLMs), evaluating both performance and fairness on clinical interview transcripts from the Distress Analysis Interview Corpus/Wizard-of-Oz (DAIC-WOZ). To mitigate bias, fairness-aware loss functions are applied to DNN-based models, while in-context learning with varied prompt framing and shot counts is explored for LLMs. Results indicate that LLMs outperform DNN-based models in depression classification, particularly for underrepresented groups such as Hispanic participants. LLMs also exhibit reduced gender bias compared to DNN-based embeddings, though racial disparities persist. Among fairness-aware techniques for mitigating bias in DNN-based embeddings, the worst-group loss, which is designed to minimize loss for the worst-performing demographic group, achieves a better balance between performance and fairness. In contrast, the fairness-regularized loss minimizes loss across all groups but performs less effectively. In LLMs, guided prompting with ethical framing helps mitigate gender bias in the 1-shot setting. However, increasing the number of shots does not lead to further reductions in disparities. For race/ethnicity, neither prompting strategy nor increasing in -shot learning effectively reduces disparities.

Paper Structure

This paper contains 17 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Description
  4. Methodology
  5. Language-based Depression Detection
  6. Bias Mitigation in DNN-based Embeddings
  7. Worst-Group Loss
  8. Fairness-Regularized Loss
  9. Bias Mitigation in LLMs
  10. Evaluation Metrics
  11. Results
  12. Depression Detection Bias in DNN-Embeddings and LLMs
  13. Bias Mitigation in DNN-Embeddings
  14. Bias Mitigation in LLMs
  15. Cost-benefit analysis
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Balanced accuracy by gender for 1, 3, and 5-shot learning across unguided prompting (UP), guided prompting (GP), and GP with ethical framing (GP-EF).