Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs

TL;DR

This work addresses multi-objective hyperparameter selection under reliability constraints by modeling the hyperparameter space as a reliability graph (RG), a directed acyclic graph that encodes expected reliability relationships. RG-PT learns the RG from prior information and data using Bradley-Terry ranking and non-negative Lasso, and then applies DAGGER-based FDR-controlled multiple hypothesis testing to select reliable hyperparameters while optimizing auxiliary objectives. The method provides formal FDR guarantees and demonstrates improved efficiency and shorter, reliable prompts across language-model prompting tasks and a sequence-to-sequence translation task, outperforming Learn-Then-Test (LTT) and Pareto Testing (PT). By exploiting structured dependencies among hyperparameters, RG-PT offers practical improvements for prompt engineering and other discrete hyperparameter settings in AI systems, with potential applicability to broader multi-objective model calibration problems.

Abstract

The selection of hyperparameters, such as prompt templates in large language models (LLMs), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can be inferred from prior information and held-out data -- e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees or are unable to incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objective hyperparameter selection framework that maintains formal reliability guarantees in terms of false discovery rate (FDR), while accounting for known relationships among hyperparameters via a directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such as learn-then-test (LTT) and Pareto testing (PT) through a more efficient exploration of the hyperparameter space.

Figures (11)

• Figure 1: Illustrative example for prompt engineering in LLM-based sentiment analysis: (a) Prompt template candidates in set $\Lambda$ have expected reliability levels that can be arranged on a reliability graph (RG), so that each parent prompt template is expected to be more reliable than its child prompts; (b) Distribution of the length of the shortest prompt templates identified by LTT angelopoulos2021learn, PT laufer2022efficiently, and the proposed RG-PT for the Stanford Sentiment Treebank dataset socher2013recursive (see Sec. \ref{['sec:prompt_engineering']} for details).
• Figure 2: Illustration of the main steps of RG-PT: ① Estimate the hyperparameters $\Lambda_\text{OPT}$ lying on the Pareto front pf problem (\ref{['sec:problem_formulation']}); ② Build the RG over the selected hyperparameters $\Lambda_\text{OPT}$; ③ Apply an FDR-controlling MHT procedure, DAGGER, to the RG to obtain the selected set $\hat{\Lambda}_\mathcal{Z}\subseteq \Lambda_\text{OPT}$.
• Figure 3: Distribution of the length of the shortest prompt templates identified by LTT angelopoulos2021learn, PT laufer2022efficiently, and the proposed RG-PT for (a) the Semantic Textual Similarity Benchmark dataset cer2017semeval and (b) the Word-in-Context dataset pilehvar2018wic.
• Figure 4: Test ROUGE-L scores achieved by LTT, PT, and RG-PT methods as a function of the target reliability value for the BLEU score.
• Figure 5: Illustration of the DAGGER algorithm's operation to control the FDR at $\delta = 0.1$. At each step, a hyperparameter is tested, starting from the root nodes and progressing level by level through the DAG. The testing thresholds $\delta_i$ are computed for each hyperparameter $\lambda_i$ using the step-up procedure in (\ref{['eq:reshaped-step-up']}) and (\ref{['eq:stepup']}), using the identity function as $\beta( \cdot )$. The p-value of each hyperparameter is compared against its respective threshold $\delta_i$ to assess the reliability of $\lambda_i$.

Theorems & Definitions (2)

• Proposition 3.1
• proof