Tailoring Self-Rationalizers with Multi-Reward Distillation

TL;DR

MaRio addresses the limitation that self-rationalization quality in small language resources is often overlooked in favor of downstream accuracy by introducing multi-reward distillation. It extends Quark with multiple reward tokens to train compact LMs to generate plausible, diverse, and consistent rationales while also improving task performance, demonstrated across five challenging QA datasets. The approach leverages GPT-3 rationales for supervision and uses carefully designed rewards—plausibility via Vera, consistency via rationale-conditioned predictions, diversity via n-gram uniqueness, and task-correctness—to steer rationale generation, with human studies confirming superior human-perceived quality. While MaRio substantially narrows the gap to large LMs on several tasks and provides a practical pathway for efficient reasoning, it also highlights the need for robust reward design and extended properties (e.g., factuality, completeness) to mitigate reward hacking and further enhance human utility.

Abstract

Large language models (LMs) are capable of generating free-text rationales to aid question answering. However, prior work 1) suggests that useful self-rationalization is emergent only at significant scales (e.g., 175B parameter GPT-3); and 2) focuses largely on downstream performance, ignoring the semantics of the rationales themselves, e.g., are they faithful, true, and helpful for humans? In this work, we enable small-scale LMs (approx. 200x smaller than GPT-3) to generate rationales that not only improve downstream task performance, but are also more plausible, consistent, and diverse, assessed both by automatic and human evaluation. Our method, MaRio (Multi-rewArd RatIOnalization), is a multi-reward conditioned self-rationalization algorithm that optimizes multiple distinct properties like plausibility, diversity and consistency. Results on five difficult question-answering datasets StrategyQA, QuaRel, OpenBookQA, NumerSense and QASC show that not only does MaRio improve task accuracy, but it also improves the self-rationalization quality of small LMs across the aforementioned axes better than a supervised fine-tuning (SFT) baseline. Extensive human evaluations confirm that MaRio rationales are preferred vs. SFT rationales, as well as qualitative improvements in plausibility and consistency.

Figures (7)

• Figure 1: Our proposed approach, MaRio. While existing self-rationalizing pipelines require exorbitantly large LMs that are used to primarily improve task performance, MaRio is a small LM that is initially distilled from rationales generated by GPT-3, following by multi-reward training that improves its rationale quality w.r.t three properties: plausibility, diversity and consistency.
• Figure 2: MaRio pipeline.MaRio uses rationales generated by a larger LM like GPT-3 as initial supervision, and uses rewards corresponding to three rationale properties: Plausibility, Diversity and Consistency, to improve self-rationalization of smaller LMs like T5-large.
• Figure 3: Results of human studies comparing MaRio with Sft. Here, we plot the % of instances in the test set wherein annotators prefer MaRio, Sft, both or none, with respect to Preference, Plausibility and Consistency. We find that human annotators vastly prefer MaRio's rationales, and also find them to be much more plausible and consistent.
• Figure 4: Reference Large LMs vs. MaRio Results: Here, we show the comparison of Avg. NRG values w.r.t the LM size (in the order of billion parameters) for all the datasets.
• Figure 5: Optimizing properties with Quark (top) and MaRio (bottom)