Comparative Analysis of Unsupervised and Supervised Autoencoders for Nuclei Classification in Clear Cell Renal Cell Carcinoma Images

TL;DR

The paper addresses automated, fine-grained ccRCC nuclei grading by learning latent representations with autoencoders and leveraging supervised signals. It introduces a classifier-augmented discriminative autoencoder (CDAE) whose CNN variant, optimized via neural architecture search with the Bhattacharyya distance ($D_B$) and subsequently by $F1$-score, yields strong latent separation and high classification accuracy. Compared with unsupervised AE variants and the CHR-Network, the CDAE-CNN approach achieves superior performance on a balanced TCGA-derived dataset, particularly for the challenging grades 2 and 3. The work demonstrates the benefit of integrating a classifier branch into AEs and points to semi-supervised extensions to reduce labeling needs in histopathology grading.

Abstract

This study explores the application of supervised and unsupervised autoencoders (AEs) to automate nuclei classification in clear cell renal cell carcinoma (ccRCC) images, a diagnostic task traditionally reliant on subjective visual grading by pathologists. We evaluate various AE architectures, including standard AEs, contractive AEs (CAEs), and discriminative AEs (DAEs), as well as a classifier-based discriminative AE (CDAE), optimized using the hyperparameter tuning tool Optuna. Bhattacharyya distance is selected from several metrics to assess class separability in the latent space, revealing challenges in distinguishing adjacent grades using unsupervised models. CDAE, integrating a supervised classifier branch, demonstrated superior performance in both latent space separation and classification accuracy. Given that CDAE-CNN achieved notable improvements in classification metrics, affirming the value of supervised learning for class-specific feature extraction, F1 score was incorporated into the tuning process to optimize classification performance. Results show significant improvements in identifying aggressive ccRCC grades by leveraging the classification capability of AE through latent clustering followed by fine-grained classification. Our model outperforms the current state of the art, CHR-Network, across all evaluated metrics. These findings suggest that integrating a classifier branch in AEs, combined with neural architecture search and contrastive learning, enhances grading automation in ccRCC pathology, particularly in detecting aggressive tumor grades, and may improve diagnostic accuracy.

Figures (3)

• Figure 1: Process of preparing nuclei patches for the AE. From left to right: nuclei are segmented using the instance map, enclosed within uniform bounding boxes, cropped, and then background is removed by reapplying the instance map to isolate nuclei within the patches.
• Figure 2: Visualization of the first three PCA components for training results of the highest-performing latent space embeddings optimized for Bhattacharyya distance. (a) AE. (b) CDAE-CNN. Note that Grade 4 refers to Non-Tumorous in this context.
• Figure 3: Pair plots of the first PCA elements for the highest performance AE latent space embeddings. (a) The train results of CDAE-CNN for the highest performance model optimized for Bhattacharyya distance. (b) The train results of CDAE-CNN for the highest performance model optimized for F1-score.