Efficient Hyper-Parameter Search for LoRA via Language-aided Bayesian Optimization

TL;DR

This work tackles the costly hyperparameter optimization problem for LoRA in fine-tuning large language models. It introduces a language-aided Bayesian optimization framework that maps discrete LoRA hyperparameters into a continuous embedding via domain-aware prompting, a learnable token, and a projection layer, enabling an effective deep-kernel GP surrogate. A proxy training evaluation on data subsets dramatically reduces computation while preserving correlation with full-data performance, allowing about 30 optimization iterations to outperform ~45,000 naive evaluations by over 20% in several tasks and models. The method generalizes across LoRA variants and architectures, reveals new empirical guidance on hyperparameter configurations, and offers a plug-and-play, transferable HPO baseline for broader fine-tuning strategies.

Abstract

Fine-tuning Large Language Models (LLMs) with Low-Rank Adaptation (LoRA) enables resource-efficient personalization or specialization, but it comes at the expense of additional hyperparameter tuning. Although LoRA makes fine-tuning efficient, it is highly sensitive to the choice of hyperparameters, and exhaustive hyperparameter search is still computationally very demanding. To address these challenges, we propose a framework that integrates the domain knowledge of pre-trained LLMs into Bayesian Optimization (BO) to efficiently search for LoRA hyperparameters. To leverage the informed knowledge of LLMs, we repurpose LLMs as a discrete-to-continuous mapping to link the hyperparameters and their domain knowledge with a continuous vector space, where BO is conducted. We design and control the mapping by language prompting, where we provide a domain-aware textual prompt describing the relationships among hyperparameters and their respective roles; thereby, we explicitly inject domain knowledge about LoRA into the LLM in natural language. Also, we model the residual information that is hard to linguistically describe in the prompt with an additional learnable token. This aids BO to sample more high-performing hyperparameters. In addition, by leveraging the observation of the strong correlation between the respective performance obtained from full and subset training datasets in LoRA training regimes, we introduce proxy training and evaluation with a data subset. This further increases the efficiency of our method. We demonstrate that our hyperparameter found with only about 30 iterations achieves more than 20% performance improvement over standard hyperparameters found from about 45,000 combinations.

Figures (1)

• Figure 1: Qualitative analysis of embedding space evolution using our components. We illustrate how the embedding space evolves with our proposed components: (a) shows the embedding space from a frozen LLM $\phi$; (b) shows the space when a projection layer $P(\cdot;\theta)$ is added to the frozen LLM; and (c) shows the space when both the projection layer and the learnable token $\psi$ are employed. The trajectories in each figure indicate optimization paths across steps, shown in arrow sequence. These results suggest that incorporating the projection layer and learnable token produces a smoother, more structured embedding space suited for BO, thereby enabling efficient optimization.