Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML

Ernesto L. Estevanell-Valladares, Suilan Estevez-Velarde, Yoan Gutiérrez, Andrés Montoyo, Ruslan Mitkov

TL;DR

XAutoLM introduces a meta-learning–augmented AutoML framework that unifies model selection and hyperparameter optimization for LM fine-tuning under compute and Green AI constraints. It builds an experience store of past pipeline evaluations and uses task and system meta-features to bias a probabilistic search, enabling rapid discovery of high-utility configurations. Across six benchmarks (four classification, two QA), warm-start priors improve Pareto efficiency and reduce evaluation time by up to several-fold, while decreasing error rates and showing meaningful cross-task transfer; results are supported by comparisons against zero-shot baselines and memory baselines. By releasing both the framework and the experience store, the work promotes reproducible, resource-efficient NLP fine-tuning and advances practical Green AI techniques.

Abstract

Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimization, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and hyperparameter optimization (HPO) task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward valuable configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimizer's peak F1 on five of six tasks, cuts mean evaluation time of pipelines by up to 4.5x, reduces search error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyze resource-efficient, Green AI fine-tuning in the NLP community.

XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML

TL;DR

XAutoLM introduces a meta-learning–augmented AutoML framework that unifies model selection and hyperparameter optimization for LM fine-tuning under compute and Green AI constraints. It builds an experience store of past pipeline evaluations and uses task and system meta-features to bias a probabilistic search, enabling rapid discovery of high-utility configurations. Across six benchmarks (four classification, two QA), warm-start priors improve Pareto efficiency and reduce evaluation time by up to several-fold, while decreasing error rates and showing meaningful cross-task transfer; results are supported by comparisons against zero-shot baselines and memory baselines. By releasing both the framework and the experience store, the work promotes reproducible, resource-efficient NLP fine-tuning and advances practical Green AI techniques.

Abstract

Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimization, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and hyperparameter optimization (HPO) task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward valuable configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimizer's peak F1 on five of six tasks, cuts mean evaluation time of pipelines by up to 4.5x, reduces search error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyze resource-efficient, Green AI fine-tuning in the NLP community.

Paper Structure

This paper contains 45 sections, 9 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposal
  4. Background
  5. Process Overview
  6. Experience Store
  7. Meta-Features
  8. Warm-Start optimization
  9. Task Similarity.
  10. Utility Score.
  11. Weighted Sum.
  12. Linear Front.
  13. Logarithmic Front.
  14. Experimentation
  15. Hardware.
  16. ...and 30 more sections

Figures (10)

  • Figure 1: Ratio of discovered pipelines outperforming the Zero-shot baseline in Text Classification and QA.
  • Figure 2: Pareto Fronts discovered by the different Priors on SST2 (a) and SQUAD (b).
  • Figure 3: Distance between Text Classification Tasks according to their meta-features (Section \ref{['sec:meta-features']}).
  • Figure 4: Initial probability distributions for fine-tuning methods on LIAR.
  • Figure 5: Initial probability distributions for fine-tuning methods on SST2
  • ...and 5 more figures