Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale

TL;DR

Reflect introduces an inference-time, data-free framework for constitutional alignment that uses in-context reasoning to align LLM outputs with natural-language principles. It combines a constitution-conditioned base generation with post-generation self-evaluation, critique, and revision to achieve high principle conformance across prompts, constitutions, and model sizes, while maintaining factual accuracy. The approach also yields synthetic data for offline fine-tuning (SFT and DPO) and demonstrates low computational overhead, tail-risk reduction, and safety advantages without unaligning models. Evaluation against human judgments and multiple datasets shows robust improvements in tail safety and general applicability to diverse deployment contexts, making Reflect a scalable and transparent tool for pluralistic AI alignment. The work also discusses practical considerations for safety-critical use and avenues for future refinements via targeted finetuning and broader principle exploration.

Abstract

The constitutional framework of alignment aims to align large language models (LLMs) with value-laden principles written in natural language (such as to avoid using biased language). Prior work has focused on parameter fine-tuning techniques, such as reinforcement learning from human feedback (RLHF), to instill these principles. However, these approaches are computationally demanding, require careful engineering and tuning, and often require difficult-to-obtain human annotation data. We propose \textsc{reflect}, an inference-time framework for constitutional alignment that does not require any training or data, providing a plug-and-play approach for aligning an instruction-tuned model to a set of principles. \textsc{reflect} operates entirely in-context, combining a (i) constitution-conditioned base response with post-generation (ii) self-evaluation, (iii)(a) self-critique, and (iii)(b) final revision. \textsc{reflect}'s technique of explicit in-context reasoning over principles during post-generation outperforms standard few-shot prompting and provides transparent reasoning traces. Our results demonstrate that \textsc{reflect} significantly improves LLM conformance to diverse and complex principles, including principles quite distinct from those emphasized in the model's original parameter fine-tuning, without sacrificing factual reasoning. \textsc{reflect} is particularly effective at reducing the rate of rare but significant violations of principles, thereby improving safety and robustness in the tail end of the distribution of generations. Finally, we show that \textsc{reflect} naturally generates useful training data for traditional parameter fine-tuning techniques, allowing for efficient scaling and the reduction of inference-time computational overhead in long-term deployment scenarios.

Figures (5)

• Figure 1: Given an input query and a constitution of principles, Reflect first generates a constitution-conditioned base response. It then self-evaluates the response before performing critique and revision to generate the final, improved response.
• Figure 2: Likert Score Rankings by Individual Principle before and after Reflect. Likert scores (1--5, higher is better) show principle-level alignment for CCBase (orange) and Reflect (blue) across three models and two datasets. Reflect achieves consistent improvements across most principles.
• Figure 3: Principle Violation Rate (%) before and after Reflect. Principle Violation Rate is defined as the percentage of responses that violate a given principle. (a) Mean Principle Violation Rate Across All Principles: aggregate performance by model and dataset. (b) Principle Violation Rate by Individual Principle: detailed breakdown for each model across both datasets. SafeRLHF contains 12 principles ($P_1, \ldots, P_{12}$) and HH-RLHF contains 10 principles ($P_1, \ldots, P_{10}$).
• Figure 4: The two primary constitutions we used in our experiments.
• Figure :