Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Anchored Alignment for Self-Explanations Enhancement

Luis Felipe Villa-Arenas, Ata Nizamoglu, Qianli Wang, Sebastian Möller, Vera Schmitt

TL;DR

This work introduces a methodology for alignment designed to enhance the ability of large language models to articulate their reasoning even in the absence of annotated rationale explanations, and presents a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs.

Abstract

In this work, we introduce a methodology for alignment designed to enhance the ability of large language models (LLMs) to articulate their reasoning (self-explanation) even in the absence of annotated rationale explanations. Our alignment methodology comprises three key components: explanation quality assessment, self-instruction dataset generation, and model alignment. Additionally, we present a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs by categorizing model outputs into three groups: consistently correct, consistently incorrect, and variable. By applying tailored strategies to each category, we enhance the effectiveness of Direct Preference Optimization (DPO). Our experimental results demonstrate that this approach significantly improves explanation quality while maintaining accuracy compared to other fine-tuning strategies.

Anchored Alignment for Self-Explanations Enhancement

TL;DR

This work introduces a methodology for alignment designed to enhance the ability of large language models to articulate their reasoning even in the absence of annotated rationale explanations, and presents a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs.

Abstract

In this work, we introduce a methodology for alignment designed to enhance the ability of large language models (LLMs) to articulate their reasoning (self-explanation) even in the absence of annotated rationale explanations. Our alignment methodology comprises three key components: explanation quality assessment, self-instruction dataset generation, and model alignment. Additionally, we present a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs by categorizing model outputs into three groups: consistently correct, consistently incorrect, and variable. By applying tailored strategies to each category, we enhance the effectiveness of Direct Preference Optimization (DPO). Our experimental results demonstrate that this approach significantly improves explanation quality while maintaining accuracy compared to other fine-tuning strategies.

Paper Structure

This paper contains 31 sections, 10 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. A Framework for Qualitative Assessment of Self-Explanations
  3. Quality Criteria for Effective Self-Explanations
  4. Self-Explanations Evaluation Methodology
  5. Pairwise Model Evaluation
  6. Self-Explanation Alignment with Anchor Preference Pairs
  7. Supervised Fine-Tuning without Annotated Explanations
  8. Self-Instruction Creation
  9. Preference Pairs via Anchor Selection
  10. Experiments
  11. Experimental Setup
  12. Results
  13. Impact of Supervised Fine-Tuning on Self-Explanations
  14. Analysis of Self-Aligned Models
  15. Analysis of Individual Evaluation Dimensions
  16. ...and 16 more sections

Figures (2)

  • Figure 1: Average Self-Explanation Scores per Evaluation Criterion. Average scores for each evaluation criterion used to assess self-explanations, as described in Section \ref{['sec: Quality Criteria for Effective Self-Explanations']}. The scores are provided for all evaluated models across the benchmark datasets.
  • Figure 2: Impact of Preference Pairs Category Distribution: Presents the relative improvements in accuracy (left) and $\Delta_{W-L}$ (right) between $\mathcal{M}_{\text{Anchor}}$ and $\mathcal{M}_{\text{Rank}}$ with respect to $\lambda$.