Paper
Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning
Authors
Ru Wang, Wei Huang, Qi Cao, Yusuke Iwasawa, Yutaka Matsuo, Jiaxian Guo
Abstract
Test-time reinforcement learning (TTRL) offers a label-free paradigm for
adapting models using only synthetic signals at inference, but its success
hinges on constructing reliable learning signals. Standard approaches such as
majority voting often collapse to spurious yet popular answers. We introduce
Self-Harmony, a framework built on a simple intuition: the correct answer
should remain stable across both an original question and its paraphrase.
Self-Harmony operationalizes this by employing a single model in two
complementary roles: a Solver to produce answers and a Reframer to rephrase the
input. Based on this, we further propose a pseudo-label method: instead of
majority voting, it aggregates answer frequencies across these original and
reframed views using the harmonic mean. This is a process that naturally
selects for solutions stable under reframing, thereby avoiding the common trap
of favoring view-dependent, spurious answers. Crucially, this requires no human
supervision or auxiliary models. Across diverse reasoning benchmarks,
Self-Harmony achieves state-of-the-art results at the label-free test-time
setting, ranking first in 28 of 30 settings across multiple methods. Beyond
accuracy, it demonstrates unprecedented robustness, with zero training failures
in all experiments, underscoring its stability and reliability.
Parse Issues: 9 warnings