Reliable Classification with Conformal Learning and Interval-Type 2 Fuzzy Sets

TL;DR

This work tackles the problem of overconfident predictions by coupling conformal learning with fuzzy rule-based classification, augmented with interval-type-2 fuzzy sets to provide calibrated uncertainty for each sample. The authors introduce interval-type-2 conformal learning and rule-wise conformal learning to generate reliable prediction sets while preserving interpretability. They detail methodological components, including a fuzzy inference system with ADAPTIVE fuzzy partitions, interval arithmetic for conformal scores, and a fitness-function design that biases toward true-class nonconformity while controlling set size. Experiments on KEEL datasets show that IV-T2 approaches improve non-empty prediction coverage and rule-level insights, though at higher computational cost and with dataset-dependent performance; fitness-function adaptation significantly reduces set sizes with minimal impact on coverage. Overall, the paper demonstrates a viable path to robust, interpretable uncertainty quantification in real-world classification tasks, with potential extensions to broader uncertainty metrics and computer vision domains.

Abstract

Classical machine learning classifiers tend to be overconfident can be unreliable outside of the laboratory benchmarks. Properly assessing the reliability of the output of the model per sample is instrumental for real-life scenarios where these systems are deployed. Because of this, different techniques have been employed to properly quantify the quality of prediction for a given model. These are most commonly Bayesian statistics and, more recently, conformal learning. Given a calibration set, conformal learning can produce outputs that are guaranteed to cover the target class with a desired significance level, and are more reliable than the standard confidence intervals used by Bayesian methods. In this work, we propose to use conformal learning with fuzzy rule-based systems in classification and show some metrics of their performance. Then, we discuss how the use of type 2 fuzzy sets can improve the quality of the output of the system compared to both fuzzy and crisp rules. Finally, we also discuss how the fine-tuning of the system can be adapted to improve the quality of the conformal prediction.

Figures (5)

• Figure 1: Why use rule-wise predictions?. In this example, we are working with three classes: roadrunner, horse and zebra. If we only used class-wise predictions, we would see that the horse class is always present in the predictions, but we would not know why. If we have rule-wise predictions we can identify and even amend the spurious patterns (R1 and R2 in the example) and leave those that are working fine (R3).
• Figure 2: Evolution of prediction size and percentage of valid predictions using IV-T2 fuzzy sets in the Iris dataset.
• Figure 3: Prediction size and non-empty predictions depending on the significance level desired.
• Figure 4: Prediction size and non-empty predictions depending on the significance level desired using a custom fitness function to reduce the average prediction size.
• Figure 5: Average F1 score of all the rules for T1 and IV-T2 fuzzy sets.