Importance Weighting Can Help Large Language Models Self-Improve

TL;DR

This work addresses the challenge of expensive external supervision for fine-tuning LLMs by introducing Distribution Shift Weight (DS weight) to quantify how self-generated data diverges from true data distributions. Building the IWSI framework, it combines correctness-based filtering (self-consistency) with DS weight-based screening, using a tiny valid set to estimate distribution shift and selectively training on low-DS samples. Empirical results across multiple reasoning and NLI tasks show that IWSI improves reasoning performance and is competitive with reward-model supervision, while showing that distribution shift is largely orthogonal to correctness. The approach offers a practical, data-efficient pathway to safer and more effective self-improvement of LLMs in real-world settings.

Abstract

Large language models (LLMs) have shown remarkable capability in numerous tasks and applications. However, fine-tuning LLMs using high-quality datasets under external supervision remains prohibitively expensive. In response, LLM self-improvement approaches have been vibrantly developed recently. The typical paradigm of LLM self-improvement involves training LLM on self-generated data, part of which may be detrimental and should be filtered out due to the unstable data quality. While current works primarily employs filtering strategies based on answer correctness, in this paper, we demonstrate that filtering out correct but with high distribution shift extent (DSE) samples could also benefit the results of self-improvement. Given that the actual sample distribution is usually inaccessible, we propose a new metric called DS weight to approximate DSE, inspired by the Importance Weighting methods. Consequently, we integrate DS weight with self-consistency to comprehensively filter the self-generated samples and fine-tune the language model. Experiments show that with only a tiny valid set (up to 5\% size of the training set) to compute DS weight, our approach can notably promote the reasoning ability of current LLM self-improvement methods. The resulting performance is on par with methods that rely on external supervision from pre-trained reward models.

Figures (6)

• Figure 1: The overview of IWSI. Given the unsupervised dataset $\mathcal{D}_q$, the pre-trained LLM $\mathcal{M}_L$ is first used to generate multiple candidate answers as well as the reasoning thoughts, prompted by CoT examples. Then IWSI uses majority voting to select the most consistent answer and corresponding thoughts, stored in $\mathcal{D}_c$. With the help of $\mathcal{D}_v$, IWSI calculates DS weight for each data point in $\mathcal{D}_c$. IWSI filters $\mathcal{D}_c$ into $\mathcal{D}_{ds}$ by keeping samples with the $k\%$-lowest DS weight and lastly self-trains $\mathcal{M}_L$.
• Figure 2: Accuracy results with varying $k$ values.
• Figure 3: $\sigma_{k\%}$ (in Eq. \ref{['equ:sigma']}) with varying $k$ values.
• Figure 4: Loss value distributions of the valid set samples, self-generated samples of base model (generated-base), and self-generated samples after IWSI (generated-IWSI), of gsm8k. $\mu$ and $\sigma$ denote the mean and standard deviation.
• Figure 5: The first row shows relationship between answer correctness and DSE, where along $x$-axis are DSE intervals and the $y$-axis indicates the proportion of correct answers and wrong answers. The second row are the DS weight probability density function curves with varying uncertainty threshold $u^*$ (Eq. \ref{['equ:uncertainty']}).