Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dropout Robustness and Cognitive Profiling of Transformer Models via Stochastic Inference

Antônio Junior Alves Caiado, Michael Hahsler

Abstract

Transformer-based language models are widely deployed for reasoning, yet their behavior under inference-time stochasticity remains underexplored. While dropout is common during training, its inference-time effects via Monte Carlo sampling lack systematic evaluation across architectures, limiting understanding of model reliability in uncertainty-aware applications. This work analyzes dropout-induced variability across 19 transformer models using MC Dropout with 100 stochastic forward passes per sample. Dropout robustness is defined as maintaining high accuracy and stable predictions under stochastic inference, measured by standard deviation of per-run accuracies. A cognitive decomposition framework disentangles performance into memory and reasoning components. Experiments span five dropout configurations yielding 95 unique evaluations on 1,000 samples. Results reveal substantial architectural variation. Smaller models demonstrate perfect prediction stability while medium-sized models exhibit notable volatility. Mid-sized models achieve the best overall performance; larger models excel at memory tasks. Critically, 53% of models suffer severe accuracy degradation under baseline MC Dropout, with task-specialized models losing up to 24 percentage points, indicating unsuitability for uncertainty quantification in these architectures. Asymmetric effects emerge: high dropout reduces memory accuracy by 27 percentage points while reasoning degrades only 1 point, suggesting memory tasks rely on stable representations that dropout disrupts. 84% of models demonstrate memory-biased performance. This provides the first comprehensive MC Dropout benchmark for transformers, revealing dropout robustness is architecture-dependent and uncorrelated with scale. The cognitive profiling framework offers actionable guidance for model selection in uncertainty-aware applications.

Dropout Robustness and Cognitive Profiling of Transformer Models via Stochastic Inference

Abstract

Transformer-based language models are widely deployed for reasoning, yet their behavior under inference-time stochasticity remains underexplored. While dropout is common during training, its inference-time effects via Monte Carlo sampling lack systematic evaluation across architectures, limiting understanding of model reliability in uncertainty-aware applications. This work analyzes dropout-induced variability across 19 transformer models using MC Dropout with 100 stochastic forward passes per sample. Dropout robustness is defined as maintaining high accuracy and stable predictions under stochastic inference, measured by standard deviation of per-run accuracies. A cognitive decomposition framework disentangles performance into memory and reasoning components. Experiments span five dropout configurations yielding 95 unique evaluations on 1,000 samples. Results reveal substantial architectural variation. Smaller models demonstrate perfect prediction stability while medium-sized models exhibit notable volatility. Mid-sized models achieve the best overall performance; larger models excel at memory tasks. Critically, 53% of models suffer severe accuracy degradation under baseline MC Dropout, with task-specialized models losing up to 24 percentage points, indicating unsuitability for uncertainty quantification in these architectures. Asymmetric effects emerge: high dropout reduces memory accuracy by 27 percentage points while reasoning degrades only 1 point, suggesting memory tasks rely on stable representations that dropout disrupts. 84% of models demonstrate memory-biased performance. This provides the first comprehensive MC Dropout benchmark for transformers, revealing dropout robustness is architecture-dependent and uncorrelated with scale. The cognitive profiling framework offers actionable guidance for model selection in uncertainty-aware applications.
Paper Structure (34 sections, 1 equation, 3 figures, 6 tables)

This paper contains 34 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dropout and Regularization
  4. Monte Carlo Dropout and Uncertainty Estimation
  5. Transformer Architecture and Mechanistic Interpretability
  6. Model Robustness and Reliability
  7. Cognitive Task Decomposition in Language Models
  8. Positioning Our Contribution
  9. Methodology
  10. Dataset Construction
  11. Data Sources
  12. Memory Task Construction
  13. Reasoning Task Construction
  14. Dataset Split
  15. Model Selection
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overall Accuracy Comparison (Top 5 Degradation). Performance comparison between Deterministic inference (blue) and Baseline MC Dropout (orange, 0.1 rate) across the five models exhibiting the largest overall degradation. Error bars represent the standard deviation across 100 stochastic forward passes.
  • Figure 2: Reasoning Task Performance. Reasoning accuracy remains relatively stable between deterministic and stochastic modes, with overlapping error bars indicating minimal impact of dropout on inferential capabilities.
  • Figure 3: Memory Task Performance. Memory accuracy exhibits severe degradation under MC Dropout, particularly for task-specialized models like roberta-base-squad2.