Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models

Nghia Bui, Guergana Savova, Lijing Wang

TL;DR

This work addresses the overlooked issue of random seed sensitivity in fine-tuning large language models by introducing macro (variance) and micro (consistency) metrics applied to GLUE and SuperGLUE benchmarks. It formalizes a novel consistency metric to capture instability of individual predictions across seeds and evaluates RoBERTa-large and Llama3.2-3B with LoRA fine-tuning, revealing significant seed-induced variability at both macro and micro levels. The findings highlight the need to report seed sensitivity in benchmarking, and suggest mitigation directions such as data-size considerations and stability-aware training, with practical implications for reproducibility and reliability of NLP evaluations.

Abstract

The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance.In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency, measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation.

Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models

TL;DR

This work addresses the overlooked issue of random seed sensitivity in fine-tuning large language models by introducing macro (variance) and micro (consistency) metrics applied to GLUE and SuperGLUE benchmarks. It formalizes a novel consistency metric to capture instability of individual predictions across seeds and evaluates RoBERTa-large and Llama3.2-3B with LoRA fine-tuning, revealing significant seed-induced variability at both macro and micro levels. The findings highlight the need to report seed sensitivity in benchmarking, and suggest mitigation directions such as data-size considerations and stability-aware training, with practical implications for reproducibility and reliability of NLP evaluations.

Abstract

The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance.In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency, measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation.

Paper Structure

This paper contains 19 sections, 3 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Macro Metric: Variance
  3. Micro Metric: Consistency
  4. Experimental Setup
  5. Benchmarks and pretrained models
  6. Settings
  7. Results and Discussion
  8. Macro impact
  9. Micro impact
  10. Discussion
  11. Conclusion
  12. Limitations
  13. Appendix
  14. Macro and Micro Metrics for Various NLP Tasks
  15. Data Description
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Macro and micro performance. A pretrained LLM is fine-tuned with random seed 42 and 52. The accuracy for both models is 60%, but the overlapping of individual predictions is 20%.
  • Figure 2: Correlation between training size (log scale), VAR, CON, and CCON while using RoBERTa-large. Tasks are arranged in ascending order of training size, with exact sizes detailed in Appendix \ref{['tab:data']}.
  • Figure 3: A heatmap of normalized $\zeta$ values across tasks and ten random seeds, with a darker color representing a better accuracy.
  • Figure 4: Correlation between training size (log scale), VAR, CON, and CCON while using Llama3.2-3B. Tasks are arranged in ascending order of training size, with exact sizes detailed in Appendix \ref{['tab:data']}.