TOWER: Tree Organized Weighting for Evaluating Complex Instructions

TL;DR

The paper addresses the challenge of evaluating complex instruction following in LLMs, where all aspects are not equally important. It introduces TOWER, a tree-based weighting scheme that assigns weights to instruction aspects based on their depth in a decomposition tree, with $weight = \frac{1}{level(v)}$ for a node $v$. Empirical results show that the LLM-based tree weighting aligns with human judgments (Spearman ≈ 0.72, close to human-human ≈ 0.74) and outperforms Direct Scoring and Ranking, using InFoBench as the testbed and releasing tree annotations and code. The work highlights improved interpretability and fairness in evaluating instruction-following capability and points to future work on expanding benchmarks and reducing evaluation cost.

Abstract

Evaluating the ability of large language models (LLMs) to follow complex human-written instructions is essential for their deployment in real-world applications. While benchmarks like Chatbot Arena use human judges to assess model performance, they are resource-intensive and time-consuming. Alternative methods using LLMs as judges, such as AlpacaEval, MT Bench, WildBench, and InFoBench offer improvements but still do not capture that certain complex instruction aspects are more important than others to follow. To address this gap, we propose a novel evaluation metric, \textsc{TOWER}, that incorporates human-judged importance into the assessment of complex instruction following. We show that human annotators agree with tree-based representations of these complex instructions nearly as much as they agree with other human annotators. We release tree-based annotations of the InFoBench dataset and the corresponding evaluation code to facilitate future research.

Figures (2)

• Figure 1: Humans prefer tree-based representations of complex instructions aspect questions over ranking based weighting. We distil these insights into a new metric, TOWER, which weights each individual aspect question based on its position within the tree representation.
• Figure 2: Percentage of aspects correctly addressed by model by decision tree level of the individual aspects. GPT-4-Turbo and Llama-3-70B have consistent performance until the 5th level when they begin to degrade. Mixtral performance degrades quickly after the first decision tree level, as indicated in Table \ref{['tab:model-performance']}