Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptation

TL;DR

Uncertainty-Guided Inference-Time Selection is introduced, a lightweight inference time framework that disentangles aleatoric and epistemic uncertainty directly in deep feature space and reduces compute by approximately 60 percent on MOT17 with negligible accuracy loss, enabling practical self regulating visual inference.

Abstract

Most estimators collapse all uncertainty modes into a single confidence score, preventing reliable reasoning about when to allocate more compute or adjust inference. We introduce Uncertainty-Guided Inference-Time Selection, a lightweight inference time framework that disentangles aleatoric (data-driven) and epistemic (model-driven) uncertainty directly in deep feature space. Aleatoric uncertainty is estimated using a regularized global density model, while epistemic uncertainty is formed from three complementary components that capture local support deficiency, manifold spectral collapse, and cross-layer feature inconsistency. These components are empirically orthogonal and require no sampling, no ensembling, and no additional forward passes. We integrate the decomposed uncertainty into a distribution free conformal calibration procedure that yields significantly tighter prediction intervals at matched coverage. Using these components for uncertainty guided adaptive model selection reduces compute by approximately 60 percent on MOT17 with negligible accuracy loss, enabling practical self regulating visual inference. Additionally, our ablation results show that the proposed orthogonal uncertainty decomposition consistently yields higher computational savings across all MOT17 sequences, improving margins by 13.6 percentage points over the total-uncertainty baseline.

Figures (4)

• Figure 1: Uncertainty behavior across MOT17. (a) Aleatoric uncertainty exhibits a right-skewed distribution (mean 0.36, std 0.19). (b) Validation on MOT17-11 shows strong correlation with conformity ($r=0.48$, $p<10^{-150}$), with binned averages increasing 3.27$\times$ from low to high uncertainty. (c) Epistemic uncertainty from the framework shows a broader, more uniform spread (mean 0.447, std 0.258), reflecting diverse model knowledge states. (d) Orthogonal decomposition over 21,324 detections, colored by conformity ($1-\text{IoU}$), reveals near-zero correlation between epistemic and aleatoric components ($r=0.048$), confirming their independence.
• Figure 2: Epistemic component analysis: (a) Spectral (bimodal), Repulsive (Gaussian), Gradient (uniform, highest entropy 5.67 bits), (b) flat epistemic trajectory (variation 0.08) vs. monotonic aleatoric (0.36) validates orthogonality.
• Figure 3: Adaptive model selection for MOT17-04. Top: Temporal uncertainty evolution (epistemic in red, aleatoric in blue). Bottom: Selected models color-coded by capacity (green=Nano, yellow=Small, orange=Medium, red=Large, dark red=XLarge). Policy correlates model scaling with epistemic spikes while ignoring aleatoric elevation, validating orthogonality-aware learning.
• Figure 4: Ablation study comparing our orthogonal uncertainty decomposition against total uncertainty baseline across five MOT17 sequences. Our method achieves 58.2% average computational savings vs. 44.6% for total uncertainty (+13.6% improvement). This substantial gap validates the necessity of separating aleatoric and epistemic components: total uncertainty conservatively uses larger models when combined uncertainty is high, while our method recognizes that high aleatoric uncertainty alone does not require increased model capacity.