It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisation

TL;DR

COALITION presents a trainable framework for smaller LLMs to deliberate via mutual reasoning by employing two variant SLMs trained on separate data splits, generating and refining diverse rationales, and using a controller to select variants during inference. The approach combines Multi-Mode Instruction Fine-Tuning, Task-Guided Selective Rationale Optimisation with Direct Preference Optimisation, and a controller-driven inference path to maximize end-task performance without external teacher LLM supervision. Empirical results across five datasets (math, NLI, commonsense) show up to ~5% absolute gains and robust generalization across model families (Phi3, Qwen, Mistral, LLaMA3) and parameter scales (4B–14B), with cross-communication between variants outperforming single-model self-refinement. These findings highlight practical benefits for deploying smaller models under commercial and ethical constraints, and suggest avenues for extending the framework with more variants or domain-specific refinements.

Abstract

Very large language models (LLMs) such as GPT-4 have shown the ability to handle complex tasks by generating and self-refining step-by-step rationales. Smaller language models (SLMs), typically with < 13B parameters, have been improved by using the data generated from very-large LMs through knowledge distillation. However, various practical constraints such as API costs, copyright, legal and ethical policies restrict using large (often opaque) models to train smaller models for commercial use. Limited success has been achieved at improving the ability of an SLM to explore the space of possible rationales and evaluate them by itself through self-deliberation. To address this, we propose COALITION, a trainable framework that facilitates interaction between two variants of the same SLM and trains them to generate and refine rationales optimized for the end-task. The variants exhibit different behaviors to produce a set of diverse candidate rationales during the generation and refinement steps. The model is then trained via Selective Rationale Optimization (SRO) to prefer generating rationale candidates that maximize the likelihood of producing the ground-truth answer. During inference, COALITION employs a controller to select the suitable variant for generating and refining the rationales. On five different datasets covering mathematical problems, commonsense reasoning, and natural language inference, COALITION outperforms several baselines by up to 5%. Our ablation studies reveal that cross-communication between the two variants performs better than using the single model to self-refine the rationales. We also demonstrate the applicability of COALITION for LMs of varying scales (4B to 14B parameters) and model families (Mistral, Llama, Qwen, Phi). We release the code for this work at https://github.com/Sohanpatnaik106/coalition.

Figures (4)

• Figure 1: Schematic flow of inference using COALITION which leverages two variants of the same LM. The sample is fed to a controller (step 1) to select the variant (steps 2-3) that generates a rationale (step 4). The generated rationale is then fed to the controller to select the variant (steps 5-6) to refine the rationale (step 7) that can be used to obtain the final answer (step 8).
• Figure 2: Training procedure of COALITION through Selective Rationale Optimisation (SRO). The task-instruction is fed to the two LLM variants (step 1) to generate different rationale candidates - $(\hat{\mathcal{R}}^{Gen}_{1}, \hat{\mathcal{R}}^{Gen}_{2})$. The IFT model $\mathcal{M}_{IFT}$ is used to score each candidate by estimating the likelihood ($\hat{l}_p$) of generating the ground-truth (GT) answer conditioned on the rationale (steps 3-4). The score is used to compare the rationales to determine the winning and the eliminated rationale candidates (step 5) which are used to tune the LLM through DPO (step 6). During the refine stage, a generated rationale candidate ($\hat{\mathcal{R}}^{Gen}_k$) is fed to both the variants to refine the rationale (step 7). The corresponding refined rationale candidates $(\hat{\mathcal{R}}^{Ref}_{k1}, \hat{\mathcal{R}}^{Ref}_{k2})$ are used to tune the model via SRO (step 8).
• Figure 3: Training process of Controller $\mathcal{C}$. Each LLM variant generates a rationale candidate (step 1). The variant that generates the winning rationale (step 2) is selected as the output label for $\mathcal{C}$ (steps 3-4). For the refine step, $\mathcal{C}$ is conditioned on the task instruction and rationale from the generate step, and is trained to select the LLM variant that generates the better refined rationale (steps 5-8).
• Figure 4: Proportion of samples routed by Controller to 1) Generate w/o Refine (5-10%), 2) Self-Refine (15-25%), 3) Cross-Refine (65-75%). Hence, generation with one variant and refining with other is preferred mode.