Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPRI: Aligning Large Language Models with Context-Situated Principles

Hongli Zhan, Muneeza Azmat, Raya Horesh, Junyi Jessy Li, Mikhail Yurochkin

TL;DR

SPRI introduces a two-stage framework that automatically generates context-specific guiding principles for each input and uses them to steer LLM responses with a critique-refine loop. The base model collaborates with a critic to iteratively synthesize K_final and then generate R_final, enabling real-time, input-tailored alignment with minimal human effort. Across cognitive reappraisal, instance-specific rubric generation, and synthetic data creation for SFT, SPRI matches or surpasses expert-crafted or oracle guidance while reducing human labor and cost. The work demonstrates substantial improvements in truthfulness and evaluative robustness, highlighting SPRI's potential as a scalable, adaptable approach to context-aware alignment in diverse tasks.

Abstract

Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.

SPRI: Aligning Large Language Models with Context-Situated Principles

TL;DR

SPRI introduces a two-stage framework that automatically generates context-specific guiding principles for each input and uses them to steer LLM responses with a critique-refine loop. The base model collaborates with a critic to iteratively synthesize K_final and then generate R_final, enabling real-time, input-tailored alignment with minimal human effort. Across cognitive reappraisal, instance-specific rubric generation, and synthetic data creation for SFT, SPRI matches or surpasses expert-crafted or oracle guidance while reducing human labor and cost. The work demonstrates substantial improvements in truthfulness and evaluative robustness, highlighting SPRI's potential as a scalable, adaptable approach to context-aware alignment in diverse tasks.

Abstract

Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.

Paper Structure

This paper contains 59 sections, 6 equations, 3 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scalable Oversight.
  4. Learning from Feedback.
  5. SPRI: A Scalable Alignment Framework with Minimal Human Oversight
  6. Stage I: Synthesizing Context-Situated Principles.
  7. Stage II: Generating Responses Guided by Synthesized Principles.
  8. SPRI for Complex Principles
  9. Can SPRI Guide Cognitive Reappraisals?
  10. Data.
  11. Baselines.
  12. SPRI Method.
  13. Evaluation & Criteria.
  14. Experimental Setup.
  15. Results.
  16. ...and 44 more sections

Figures (3)

  • Figure 1: Using SPRI, GPT-4o-mini can generate situated and detailed principles to guide the response to a person narrating in distress. Compared with generic rules bai2022constitutional and human-expert-crafted principles zhan_2024_reappraisal, SPRI requires minimal to no human efforts yet produces context-specific guidance for every query at hand.
  • Figure 2: Overview for SPRI, which consists of two stages: 1) producing a set of principles specifically tailored to the user's input $T$, and 2) utilizing the generated principles to guide the response to $T$. Both stages include a critique-refine process involving a separate critic model, which aims to scrutinize the fitness of the principles to $T$ and the final responses' adherence to the generated principles.
  • Figure 3: The $6$ default seed principles used in the SPRI framework.