Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-training Language Models for Arithmetic Reasoning

Marek Kadlčík, Michal Štefánik

TL;DR

This work explores the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training), and finds that models can substantially improve in both single-round (offline) and online self-training.

Abstract

Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.

Self-training Language Models for Arithmetic Reasoning

TL;DR

This work explores the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training), and finds that models can substantially improve in both single-round (offline) and online self-training.

Abstract

Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.
Paper Structure (17 sections, 2 figures, 4 tables)

This paper contains 17 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiments
  4. Offline Self-training
  5. Results
  6. Online Self-training
  7. Results
  8. Conclusions
  9. Training Details
  10. Online self-training
  11. Supervised training:
  12. Preference Optimization:
  13. Output analyses
  14. Examples of predictions
  15. id: gsm8k__aBNdImSK9ddCwuYb
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Schema of self-training that we apply to provide the model with training feedback to its predictions. In the offline variant, the model generates all predictions in a single round. In the online variant, the training data is continuously generated.
  • Figure 2: Training dynamics of online training: The fraction of training problems for which the model predicted all and none of 16 trials correctly during training of the online KTO with $\beta=0.1$. The fraction is computed from a sliding window of the last 1000 problems and the chart is smoothed for visual clarity.