Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Small but Fair! Fairness for Multimodal Human-Human and Robot-Human Mental Wellbeing Coaching

Jiaee Cheong, Micol Spitale, Hatice Gunes

TL;DR

The paper tackles fairness in ML for small multimodal datasets in wellbeing coaching, extending fairness analysis to human-human and robot-human interactions. It introduces MixFeat, a Mixup-based pre-processing data augmentation method, to counteract biases across gender and race while leveraging high-level, multimodal features. Through comprehensive experiments on AFAR-BSFT, AFAR-RC22, and AFAR-RC23, the authors show that high-level features and multimodal fusion generally improve both predictive accuracy and fairness, and that MixFeat can outperform standard oversampling baselines. The work provides practical recommendations for researchers in affective computing and HRI to adopt fairness-aware ML practices, especially when data are scarce, and emphasizes ethical considerations for deploying robot-assisted wellbeing coaching.

Abstract

In recent years, the affective computing (AC) and human-robot interaction (HRI) research communities have put fairness at the centre of their research agenda. However, none of the existing work has addressed the problem of machine learning (ML) bias in HRI settings. In addition, many of the current datasets for AC and HRI are "small", making ML bias and debias analysis challenging. This paper presents the first work to explore ML bias analysis and mitigation of three small multimodal datasets collected within both a human-human and robot-human wellbeing coaching settings. The contributions of this work includes: i) being the first to explore the problem of ML bias and fairness within HRI settings; and ii) providing a multimodal analysis evaluated via modelling performance and fairness metrics across both high and low-level features and proposing a simple and effective data augmentation strategy (MixFeat) to debias the small datasets presented within this paper; and iii) conducting extensive experimentation and analyses to reveal ML fairness insights unique to AC and HRI research in order to distill a set of recommendations to aid AC and HRI researchers to be more engaged with fairness-aware ML-based research.

Small but Fair! Fairness for Multimodal Human-Human and Robot-Human Mental Wellbeing Coaching

TL;DR

The paper tackles fairness in ML for small multimodal datasets in wellbeing coaching, extending fairness analysis to human-human and robot-human interactions. It introduces MixFeat, a Mixup-based pre-processing data augmentation method, to counteract biases across gender and race while leveraging high-level, multimodal features. Through comprehensive experiments on AFAR-BSFT, AFAR-RC22, and AFAR-RC23, the authors show that high-level features and multimodal fusion generally improve both predictive accuracy and fairness, and that MixFeat can outperform standard oversampling baselines. The work provides practical recommendations for researchers in affective computing and HRI to adopt fairness-aware ML practices, especially when data are scarce, and emphasizes ethical considerations for deploying robot-assisted wellbeing coaching.

Abstract

In recent years, the affective computing (AC) and human-robot interaction (HRI) research communities have put fairness at the centre of their research agenda. However, none of the existing work has addressed the problem of machine learning (ML) bias in HRI settings. In addition, many of the current datasets for AC and HRI are "small", making ML bias and debias analysis challenging. This paper presents the first work to explore ML bias analysis and mitigation of three small multimodal datasets collected within both a human-human and robot-human wellbeing coaching settings. The contributions of this work includes: i) being the first to explore the problem of ML bias and fairness within HRI settings; and ii) providing a multimodal analysis evaluated via modelling performance and fairness metrics across both high and low-level features and proposing a simple and effective data augmentation strategy (MixFeat) to debias the small datasets presented within this paper; and iii) conducting extensive experimentation and analyses to reveal ML fairness insights unique to AC and HRI research in order to distill a set of recommendations to aid AC and HRI researchers to be more engaged with fairness-aware ML-based research.
Paper Structure (42 sections, 1 equation, 5 figures, 7 tables)

This paper contains 42 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Robotic Mental wellbeing Coaching
  4. Fairness in Mental wellbeing
  5. ML Bias and Fairness in HRI
  6. Data Augmentation for Bias Mitigation
  7. Problem Formulation
  8. Fairness Measures
  9. Proposed Method: MixFeat
  10. Datasets and Methods
  11. Datasets
  12. The AFAR BSFT Dataset
  13. Data Collection
  14. Sensitive Groups
  15. The AFAR Robocoaching 2022 Dataset (AFAR-RC22)
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Proportion of small dataset papers accepted at ACII'21-'23 and HRI'22 - '24. Blue represents count for ACII. Green represents count for HRI.
  • Figure 2: AFAR-BSFT Dataset: interaction between the participants (on the left side), and the human wellbeing coach (on the right side).
  • Figure 3: AFAR Robocoaching 2022 Dataset (AFAR-RC22): the setting of the study. (QT) 14 participants interacted with the QT robot; (M) 12 participants interacted with the Misty II robot spitale2023robotic.
  • Figure 4: AFAR-Robo Coaching 2023 Dataset (ARC-2023): the setting of the study. 29 participants interacted with the LLM-powered QT robot spitale2023vita.
  • Figure 5: The model pipeline with our proposed data augmentation technique: MixFeat. After extracting the high-level features from the dataset, we generate synthetic sample features using Equation \ref{['eqn:mixmod']}. Each modality's feature generation process is chiefly governed by their respective $\lambda \sim$ Beta(0,1) parameters.