Validating Vision Transformers for Otoscopy: Performance and Data-Leakage Effects

TL;DR

This study addresses the diagnostic challenge of ear diseases by evaluating Swin Transformer vision transformers against a CNN baseline (ResNet50v2) on a real-world otoscopic video dataset. It highlights a critical data leakage issue—train-test overlap from adjacent frames—that artificially inflated pre-correction performance, and demonstrates substantial performance declines (to approximately 83% for Swin variants and 82% for ResNet50) after proper data segregation. The findings underscore the importance of rigorous data handling and preprocessing when applying advanced architectures to medical imaging, while suggesting that vision transformers hold promise if coupled with robust data practices. The work has practical implications for clinical decision support and telemedicine, contingent on prospective validation and careful consideration of data leakage and class balance in future deployments.

Abstract

This study evaluates the efficacy of vision transformer models, specifically Swin transformers, in enhancing the diagnostic accuracy of ear diseases compared to traditional convolutional neural networks. With a reported 27% misdiagnosis rate among specialist otolaryngologists, improving diagnostic accuracy is crucial. The research utilised a real-world dataset from the Department of Otolaryngology at the Clinical Hospital of the Universidad de Chile, comprising otoscopic videos of ear examinations depicting various middle and external ear conditions. Frames were selected based on the Laplacian and Shannon entropy thresholds, with blank frames removed. Initially, Swin v1 and Swin v2 transformer models achieved accuracies of 100% and 99.1%, respectively, marginally outperforming the ResNet model (99.5%). These results surpassed metrics reported in related studies. However, the evaluation uncovered a critical data leakage issue in the preprocessing step, affecting both this study and related research using the same raw dataset. After mitigating the data leakage, model performance decreased significantly. Corrected accuracies were 83% for both Swin v1 and Swin v2, and 82% for the ResNet model. This finding highlights the importance of rigorous data handling in machine learning studies, especially in medical applications. The findings indicate that while vision transformers show promise, it is essential to find an optimal balance between the benefits of advanced model architectures and those derived from effective data preprocessing. This balance is key to developing a reliable machine learning model for diagnosing ear diseases.

Figures (14)

• Figure 1: Firefly camera used for the examination 10.
• Figure 2: Image comparison between an image of a chess board and the Laplacian image, to visually explain the Laplacian concept.
• Figure 3: A side-by-side comparison of the lowest (Left) and highest (Right) entropy frames from a given video in the dataset.
• Figure 4: Accuracy and Recall gathered from the Swin v1, Swin v2 and ResNet 50 models before addressing data leakage.
• Figure 5: F1-score and MCC gathered from the Swin v1, Swin v2 and ResNet 50 models before addressing data leakage.