Toxicity Assessment in Preclinical Histopathology via Class-Aware Mahalanobis Distance for Known and Novel Anomalies

TL;DR

An AI-based anomaly detection framework for histopathological whole-slide images (WSIs) in rodent livers from toxicology studies is introduced, demonstrating the potential of AI-driven histopathology to support preclinical workflows, reduce late-stage failures, and improve efficiency in drug development.

Abstract

Drug-induced toxicity remains a leading cause of failure in preclinical development and early clinical trials. Detecting adverse effects at an early stage is critical to reduce attrition and accelerate the development of safe medicines. Histopathological evaluation remains the gold standard for toxicity assessment, but it relies heavily on expert pathologists, creating a bottleneck for large-scale screening. To address this challenge, we introduce an AI-based anomaly detection framework for histopathological whole-slide images (WSIs) in rodent livers from toxicology studies. The system identifies healthy tissue and known pathologies (anomalies) for which training data is available. In addition, it can detect rare pathologies without training data as out-of-distribution (OOD) findings. We generate a novel dataset of pixelwise annotations of healthy tissue and known pathologies and use this data to fine-tune a pre-trained Vision Transformer (DINOv2) via Low-Rank Adaptation (LoRA) in order to do tissue segmentation. Finally, we extract features for OOD detection using the Mahalanobis distance. To better account for class-dependent variability in histological data, we propose the use of class-specific thresholds. We optimize the thresholds using the mean of the false negative and false positive rates, resulting in only 0.16\% of pathological tissue classified as healthy and 0.35\% of healthy tissue classified as pathological. Applied to mouse liver WSIs with known toxicological findings, the framework accurately detects anomalies, including rare OOD morphologies. This work demonstrates the potential of AI-driven histopathology to support preclinical workflows, reduce late-stage failures, and improve efficiency in drug development.

Figures (11)

• Figure 1: Two-step training approach of the system that performs semantic segmentation of tissue states with available training data (in-distribution (ID) data; i.e., healthy tissue and a set of common pathologies); additionally, it can detect tissue states without training examples as out-of-distribution (OOD) anomalies (i.e., rare pathologies, or artifacts). A. In Step 1, the system is trained as a conventional segmentation approach using a foundation model (DINOv2), adjusted to the known histopathology data. B. Example of a segmentation output on mouse liver stained in H&E. On this patch, alongside with healthy tissue (5), several anomaly types are detected: ballooning (1), inflammation (2), mitosis (3), and necrosis (4). Segmentation also shows a void in the tissue (6), resulting from cross-sectioning the vascular system. C. In Step 2, the trained segmentation model is used to encode images pixelwise into descriptive features, which allow to detect anomalies. For each ID histopathology class, the distribution in the feature space is obtained and the mean (class-wise) and covariance matrix (shared over classes) of these features is used to obtain the Mahalanobis score. We determine thresholds on the scores per class, motivated by a strong class-specific variability. At inference time we use the pixelwise scores of all classes to discriminate between known ID data and unknown OOD data using the encoded features of a test image. D. Schematic difference between the standard threshold selection strategy where all scores from the training/validation set form a single distribution and adaptive per-class selection strategy where we decompose the global score distribution into class-specific distributions and assign individual thresholds.
• Figure 2: Inference workflow for segmentation and anomaly detection. WSIs (H&E-stained mouse liver) are divided into patches and processed by an encoder to extract spatially resolved feature representations ($\mathbf{h}$). These features are passed to a classifier to predict the most likely class ($c_i$), and the outputs are averaged across spatial shifts for increased robustness. For each predicted class, the Mahalanobis distance ($s_{Maha+}$) between the features and the estimated class mean is computed using the estimated covariance matrix. Based on the class-specific threshold ($\tau_{i,p}$), a spatial location in a patch is either confirmed as in-distribution (ID) with class $c_i$ or flagged as out-of-distribution (OOD) when the score is below the threshold.
• Figure 3: Schematic representation of spatial averaging procedure applied to improve robustness during inference and mitigate boundary artifacts.
• Figure 4: Extended confusion matrix for unified segmentation and anomaly detection.
• Figure 5: Distribution of the mean anomaly score per tile for healthy (no damage model was induced) tissue samples. The samples with the highest scores are the most normal. The outliers with lower scores (i.e., rated as more anomalous) are samples with low glycogen storage as well as samples at the tissue boundary. The samples lying within the central 99% quantile interval exhibit the gradient transition from normal to low glycogen storage.