Dynamic Bayesian Network Modelling of User Affect and Perceptions of a Teleoperated Robot Coach during Longitudinal Mindfulness Training

TL;DR

A dynamic Bayesian network (DBN) is presented to capture the longitudinal interactions participants had with a teleoperated robot coach (RC) delivering mindfulness sessions to facilitate learning of conditional dependency structure between variables in the longitudinal data and offers inferences and conceptual understanding which are not possible through other regression methodologies.

Abstract

Longitudinal interaction studies with Socially Assistive Robots are crucial to ensure that the robot is relevant for long-term use and its perceptions are not prone to the novelty effect. In this paper, we present a dynamic Bayesian network (DBN) to capture the longitudinal interactions participants had with a teleoperated robot coach (RC) delivering mindfulness sessions. The DBN model is used to study complex, temporal interactions between the participants self-reported personality traits, weekly baseline wellbeing scores, session ratings, and facial AUs elicited during the sessions in a 5-week longitudinal study. DBN modelling involves learning a graphical representation that facilitates intuitive understanding of how multiple components contribute to the longitudinal changes in session ratings corresponding to the perceptions of the RC, and participants relaxation and calm levels. The learnt model captures the following within and between sessions aspects of the longitudinal interaction study: influence of the 5 personality dimensions on the facial AU states and the session ratings, influence of facial AU states on the session ratings, and the influences within the items of the session ratings. The DBN structure is learnt using first 3 time points and the obtained model is used to predict the session ratings of the last 2 time points of the 5-week longitudinal data. The predictions are quantified using subject-wise RMSE and R2 scores. We also demonstrate two applications of the model, namely, imputation of missing values in the dataset and estimation of longitudinal session ratings of a new participant with a given personality profile. The obtained DBN model thus facilitates learning of conditional dependency structure between variables in the longitudinal data and offers inferences and conceptual understanding which are not possible through other regression methodologies.

Figures (9)

• Figure 1: Participants interacting with the teleoperated mindfulness robot coach.
• Figure 2: Teleoperation setup, where an experienced human coach is remotely teleoperating Pepper through real-time pose replication. The HMD worn by the human coach enables them to see what the robot sees in the adjacent room. An audio pipeline is also created to enable the coach to have a conversation with the participants.
• Figure 3: AU intensities corresponding to the 18 AUs extracted for a selection of $4$ frames are shown here. The intensities ranged from $0$ (not present), $1$ (present at minimum intensity), $5$ (present at maximum intensity), with continuous values in between. We only considered AU intensities greater than $1$ as it corresponds the minimum intensity value, if the AU is present. AUs active for Frame $12$ are Brow Lowerer (AU04), Lid Tightener (AU07) and Dimpler (AU14); for Frame $732$ are Brow Lowerer (AU04); for Frame $936$ are Brow Lowerer (AU04), Lips part (AU25) and Jaw Drop (AU26); for Frame $1332$ are Brow Lowerer (AU04), Cheek Raiser (AU06), Lip Corner Puller (AU12) and Dimpler (A14).
• Figure 4: Plot of BIC (in orange) and AIC (in green) scores against number of clusters $AU\_med$. BIC scores were found to be minimum for the cluster number $K=4$.
• Figure 5: Plot of BIC (in orange) and AIC (in green) scores against number of clusters for $AU\_int$. BIC scores were found to be minimum for the cluster number $K=4$.