Tandem Training for Language Models

TL;DR

This work addresses the challenge that highly capable AI reasoning can become opaque to weaker actors. It introduces tandem training, an RL-based paradigm where a senior model and a frozen junior alternately generate next tokens, with rewards contingent on the junior's ability to continue the solution, thereby enforcing handoff robustness $M_{sen}$ vs. $M_{jun}$. Across GSM8K math problems, tandem training dramatically reduces jargon while keeping accuracy above the junior baseline, even when senior and junior differ in skill and language. The approach offers a practical path toward auditable, human-and-AI collaborative systems and paves the way for safer, multi-agent interactions with scalable oversight.

Abstract

As language models continue to rapidly improve, we can expect their actions and reasoning to become difficult or impossible for weaker agents and humans to follow, undermining interpretability and oversight. With an eye on long-term futures, we pursue methods that encourage models to produce solutions that remain intelligible to weaker collaborators. We formalize intelligibility as handoff robustness: a strong model's solution is intelligible to a weaker model if randomly handing off control to the weaker model along the solution path does not cause failure. Building on this criterion, we introduce tandem training for language models, a reinforcement learning (RL) paradigm in which rollout tokens are intermittently and randomly sampled from a frozen weak model rather than the strong model being trained. Because rollouts succeed only when the strong model's actions and reasoning process can be continued by the weak model -- when the two can co-construct a successful solution -- optimizing standard RL objectives with tandem training implicitly incentivizes both correctness and intelligibility. In the GSM8K math reasoning task, tandem training reliably teaches models to abandon jargon and adapt their language to weaker partners while keeping task accuracy high. Our results demonstrate a promising route to building AI systems that remain auditable by weaker agents, with implications for human--AI collaboration and multi-agent communication.

Figures (4)

• Figure 1: Tandem training vs. standard RL. Standard RL (left) only encourages correct solutions, regardless of whether they are intelligible to other models or humans, which raises concerns about interpretability and oversight. We define intelligibility via handoff robustness, positing that a partial solution is intelligible to another agent if that agent could continue the solution, at least for a few steps, without derailing it into failure. Tandem training(right) encourages handoff robustness, and thus intelligibility, by letting the trained model ("senior") generate solutions together with a frozen (typically weaker) "junior" model, taking turns randomly. Since rewards are high only if neither senior nor junior made a critical mistake, the senior is encouraged to generate solutions that can be correctly continued (and thus, by our definition, understood) by the junior.
• Figure 2: Column 1: Results for skill disparity setting, where tandem-trained senior is GSM8K specialist, and frozen junior is Llama-2 prompted in English. Columns 2--4: Results for skill and language disparity setting, where junior is Llama-2 prompted in non-English languages (one per column: German, French, Bulgarian, Serbian). Row 1: Use of notational jargon ($\ll$$\gg$). Row 2: Use of linguistic jargon (English). Row 3: Use of junior language. Row 4: Accuracy. Curves: Tandem-trained senior (with 95% CIs). Shaded bands: Frozen junior (95% CIs). Takeaway: Tandem training reduces senior model jargon while keeping senior accuracy above junior level.
• Figure 3: Results for language disparity setting, where tandem-trained senior (rows) and frozen junior (columns) are Llama-2 models prompted in different languages. Plots show senior's language use when applying checkpoints from tandem training to GSM8K test data. (Senior and junior accuracy shown in Appendix \ref{['app:Supplemental results']}, Fig. \ref{['fig:results_lang_disparity__accuracy']}.) Takeaway: Tandem training reduces senior jargon (use of senior's input language) without driving accuracy below junior level.
• Figure 4: Senior and junior accuracy for language disparity setting, where tandem-trained senior (rows) and frozen junior (columns) are Llama-2 models prompted in different languages.