Respiratory Status Detection with Video Transformers

Abstract

Recognition of respiratory distress through visual inspection is a life saving clinical skill. Clinicians can detect early signs of respiratory deterioration, creating a valuable window for earlier intervention. In this study, we evaluate whether recent advances in video transformers can enable Artificial Intelligence systems to recognize the signs of respiratory distress from video. We collected videos of healthy volunteers recovering after strenuous exercise and used the natural recovery of each participants respiratory status to create a labeled dataset for respiratory distress. Splitting the video into short clips, with earlier clips corresponding to more shortness of breath, we designed a temporal ordering challenge to assess whether an AI system can detect respiratory distress. We found a ViViT encoder augmented with Lie Relative Encodings (LieRE) and Motion Guided Masking, combined with an embedding based comparison strategy, can achieve an F1 score of 0.81 on this task. Our findings suggest that modern video transformers can recognize subtle changes in respiratory mechanics.

Figures (1)

• Figure 1: Accuracy analysis for the top performing LieRE-MGM-embedding model as a function of shortness of breath severity. The larger the difference in clip position, the more likely the model was able to accurately assess the subject’s respiratory status. A clip position of 1 is equal to 6 seconds difference in a participant's respiratory recovery course.