From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

TL;DR

A unified framework, ACT (Aligning to ConsTraints), is proposed, to automatically produce supervision signals for user alignment with constraints, and is able to enhance LMs' capability to adhere to different classes of constraints, thereby improving task performance.

Abstract

User alignment is crucial for adapting general-purpose language models (LMs) to downstream tasks, but human annotations are often not available for all types of instructions, especially those with customized constraints. We observe that user instructions typically contain constraints. While assessing response quality in terms of the whole instruction is often costly, efficiently evaluating the satisfaction rate of constraints is feasible. We investigate common constraints in NLP tasks, categorize them into three classes based on the types of their arguments, and propose a unified framework, ACT (Aligning to ConsTraints), to automatically produce supervision signals for user alignment with constraints. Specifically, ACT uses constraint verifiers, which are typically easy to implement in practice, to compute constraint satisfaction rate (CSR) of each response. It samples multiple responses for each prompt and collect preference labels based on their CSR automatically. Subsequently, ACT adapts the LM to the target task through a ranking-based learning process. Experiments on fine-grained entity typing, abstractive summarization, and temporal question answering show that ACT is able to enhance LMs' capability to adhere to different classes of constraints, thereby improving task performance. Further experiments show that the constraint-following capabilities are transferable.

Figures (8)

• Figure 1: Each user instruction contains one or more constraints. The same task may be associated with different constraints depending on user intents, whereas different tasks may share similar constraints.
• Figure 2: An example of fine-grained entity typing with label option and label hierarchy constraints. A feasible response must satisfy both constraints.
• Figure 3: Overview of ACT. ACT utilizes automatic constraint verifiers, which are typically easy to implement in practice, to assess how well a response satisfies the constraints specified in the instruction. It samples two or more responses (e.g., RA and RB) for each prompt. Then, it computes the constraint satisfaction rate (CSR) of each response and assigns the preference label to each response pair based on their CSR (e.g., RA is better than RB). The preference labels serve as supervision signals for LM alignment.
• Figure 4: Results on fine-grained entity typing with $f(y)$ constraint. ACT, using supervision signals from automatic constraint verifiers, achieves performance close to that of Finetuning on the same amount of labeled data.
• Figure 5: Average CSR of raw responses on fine-grained entity typing. Label Option constraint limits the candidate set of entity types. Label Hierarchy constraint requires the answer to follow the hierarchy between coarse- and fine-grained entity types. A correct answer must satisfy Both constraints. ACT achieves CSR comparable to that of Finetuning.