The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

TL;DR

This work investigates why performance on NLP analysis datasets (NLI and RC) is unstable across seeds and training runs. It demonstrates that inter-example correlations drive the majority of variance, supported by a variance-decomposition framework that isolates independent data variance from inter-data covariance. The authors show that correlations between analysis sets and standard dev sets are low, undermining common model-selection practices and reproducibility. They propose reporting decomposed variance components and advocate for dataset diversification and stronger modeling biases to improve stability and interpretability of analysis results.

Abstract

We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. Our code is publicly available at: https://github.com/owenzx/InstabilityAnalysis

Figures (4)

• Figure 1: The trajectories of BERT performance on SNLI, MNLI-m, HANS mccoy2019right, and the Numerical subcategory of the Stress Test dataset naik2018stress (from the topmost line to the bottom, respectively). The solid lines represent the means of ten runs and the shadow area indicates a distance within a standard deviation from the means. The two dashed lines show the trajectories of one single run for MNLI-m and Numerical Stress Test using the same model.
• Figure 2: The results of BERT, RoBERTa, and XLNet on all datasets with 10 different random seeds. Large variance can be seen at certain analysis datasets (e.g. STR-NU, HANS, etc.) while results on standard validation sets are always stable.
• Figure 3: Spearman's correlations for different datasets showing the low correlation between standard datasets (i.e., MNLI-m, MNLI-mm, and SNLI) and all the other analysis datasets.
• Figure 4: The two heatmaps of inter-example correlations matrices for both MNLI and HANS. Each point in the heatmap represents the Spearman's correlation between the predictions of an example-pair.