Auditory-Tactile Congruence for Synthesis of Adaptive Pain Expressions in RoboPatients

TL;DR

It is demonstrated that modulation of pitch and amplitude is critical for achieving perceptually coherent auditory tactile mappings in robotic pain simulation and supports the development of high fidelity robotic patient simulators and provides a platform for studying multidimensional representations of pain in embodied robotic systems.

Abstract

Misdiagnosis can result in delayed treatment and patient harm. Robotic patient simulators (robopatients) provide a controlled framework for training and evaluating clinicians in rare and complex cases. We investigate auditory tactile congruence in the synthesis of adaptive vocal pain expressions for robopatients. The system generates pain vocalizations in response to tactile stimuli applied during abdominal palpation. Haptic input is captured through an abdominal phantom and processed using an internal palpation-to-pain mapping model that drives acoustic output. To evaluate perceptual congruence between palpation force and synthesized pain expressions, we conducted a study comprising 7,680 trials with 20 participants. Participants rated perceived pain intensity based solely on auditory feedback. We analyzed the influence of acoustic parameters on agreement between applied force and perceived pain. Results indicate that amplitude and pitch significantly affect perceptual agreement, independent of pain sound category. Increased palpation force was associated with higher agreement ratings, consistent with psychophysical scaling effects. Among the tested acoustic features, pitch exerted a stronger influence than amplitude on perceived congruence. These findings demonstrate that modulation of pitch and amplitude is critical for achieving perceptually coherent auditory tactile mappings in robotic pain simulation. The proposed framework supports the development of high fidelity robotic patient simulators and provides a platform for studying multidimensional representations of pain in embodied robotic systems.

Figures (8)

• Figure 1: Overview of the conceptual model for studying auditory-tactile congruence in robopatients. The system comprises three main components: the user (ideally a physician), the robopatient, and the human, providing feedback to train the robopatient. The human patients under examination may exhibit various physiological abnormalities, psychological conditions, and individual characteristics, all of which influence the dimensionality of pain expression in response to palpation. The robopatient serves as an adaptive intermediary, learning appropriate pain responses from the user and aiming to replicate human vocal pain expressions (including facial pain expressions for realism) realistically. During palpation, the robot’s abdominal phantom captures the user’s haptic input, which is recorded and processed through the robopatient’s internal palpation-pain mapping model. This conceptual model is derived from data collected from participants of diverse backgrounds during previous studies lalitharatne2022faceprotpagorn2023vocal11177232, and these studies inform a universal method for mapping palpation to pain expressions, formulated as the "auditory-tactile congruence model". Then the robot outputs corresponding pain sounds and facial expressions through MorphFace lalitharatne2021morphface. The user provides feedback on the robopatient’s generated pain responses, which were later refined by comparing palpation data with user feedback to map the relationship between palpation and the pain expressions it causes. This concept was adapted from lalitharatne2022face and modified to fit the new paradigm.
• Figure 2: Physical robot and the experiment procedure A) Robopatient setup. A scenario where the participant is palpating the robopatient is shown. B) Data flow within the robopatient setup during the experiment. The interaction between the robot and the user starts as the user palpates the abdominal phantom (Steps 1, 2). The force the user applies has to reach a certain level indicated by a progress bar next to the robot's face (Step 3). The progress bar is initially green and turns red when the required minimum amount of force is applied. Based on the palpation force, the robot generates facial expressions and sounds independently of each other (Step 4). Then the user listens to the pain sounds generated by the robot for the palpation force (Steps 5, 6) and provides feedback on whether they agree with the robot’s mapping of pain sounds to palpation.
• Figure 3: A) Median values of all participants' responses ('strongly agree'/'agree') for the amplitude (primary y axis- shown in black) and pitch (secondary y axis- shown in orange) of male and female pain sounds. B) Modes of all participants' responses for the amplitude (primary y axis- shown in black) and pitch (secondary y axis- shown in orange) of male and female pain sounds.
• Figure 4: A) The responses given by the participants (Agree/Strongly agree and Disagree/Strongly disagree) for all the trials throughout the experiment are presented as percentages. B) The response probability of each response is shown here.
• Figure 5: Violin plot visualizing the distribution of actual force responses across different target forces. The width of each violin reflects the density of actual forces, with the central line indicating the median and the shaded area showing the inter-quartile range (IQR). This visual highlights differences in actual force distributions among the four target force categories. The plots represent the force responses for male and female participants for each target force, respectively (refer to legend). The dashed black lines represent the mean actual force of each target force.