Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models

Huimin Ren, Yan Liang, Baiqiao Su, Chaobo Sun, Hengtong Lu, Kaike Zhang, Chen Wei

TL;DR

LexInstructEval tackles the challenge of evaluating fine-grained lexical instruction following in LLMs by introducing a formal grammar that encodes instructions as the $\langle \texttt{Procedure, Relation, Value} \rangle$ triplet and a transparent automated verification engine. It provides a bilingual English-Chinese benchmark assembled via a four-stage data-construction pipeline with human validation, along with open-source tooling for objective adjudication based on rigid, universal-quantification checks. Experimental results across diverse open- and closed-source models show that instruction depth, not just quantity, governs performance, with noticeable cross-lingual gaps and high agreement with expert judgments. This framework offers a scalable, low-cost means to benchmark and guide improvements in controllability and reliability of LLMs. It is poised to inform future developments in precise instruction adherence and multilingual evaluation.”

Abstract

The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated LLM-as-a-judge systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical <Procedure, Relation, Value> triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.

LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models

TL;DR

LexInstructEval tackles the challenge of evaluating fine-grained lexical instruction following in LLMs by introducing a formal grammar that encodes instructions as the triplet and a transparent automated verification engine. It provides a bilingual English-Chinese benchmark assembled via a four-stage data-construction pipeline with human validation, along with open-source tooling for objective adjudication based on rigid, universal-quantification checks. Experimental results across diverse open- and closed-source models show that instruction depth, not just quantity, governs performance, with noticeable cross-lingual gaps and high agreement with expert judgments. This framework offers a scalable, low-cost means to benchmark and guide improvements in controllability and reliability of LLMs. It is poised to inform future developments in precise instruction adherence and multilingual evaluation.”

Abstract

The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated LLM-as-a-judge systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical <Procedure, Relation, Value> triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.

Paper Structure

This paper contains 26 sections, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. 1. Introduction
  2. 2. Related Work
  3. Evaluation of LLMs.
  4. Instruction-Following Evaluation.
  5. 3. Approach
  6. 3.1. Methodology for Data Construction
  7. 3.1.1. System Design
  8. 3.1.2. Conflict Filtering
  9. 3.1.3. Difficulty Grading
  10. 3.1.4. LLM & Human Check
  11. 3.1.5. Final Dataset Composition
  12. 3.2. Automated Verification Engine
  13. 3.2.1. Element Isolation via Procedural Parsing
  14. 3.2.2. Target Identification within Isolated Elements
  15. 3.2.3. Adjudication against the Evaluation Rubric
  16. ...and 11 more sections

Figures (4)

  • Figure 1: A motivating example illustrating the core idea of LexInstructEval. Complex natural language instructions are decomposed into fine-grained, verifiable rules, which are then checked by our automated verification engine.
  • Figure 2: The overall workflow of LexInstructEval, encompassing the four-stage data construction pipeline and the three-stage automated verification engine.
  • Figure 3: Strict Accuracy of evaluated models across different difficulty levels.
  • Figure 4: Heatmap analysis of Strict Accuracy, broken down by Procedure Depth (y-axis) and Instruction Count (x-axis).