Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM2: Let Large Language Models Harness System 2 Reasoning

Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, Wai Lam

TL;DR

This work tackles the tendency of LLMs to produce undesirable outputs by leveraging a dual-process design that pairs a System 1 LLM with a System 2 process-based verifier. The verifier is trained with synthetic process-supervision data generated through a token quality exploration strategy, and it can steer generation at each token via a process score. Empirically, LLM2 improves mathematical reasoning on GSM8K and MATH across Llama3 and Qwen2.5 models, with self-consistency and self-generated data further boosting performance and demonstrating favorable latency compared to token-level decoding baselines. The approach offers a scalable way to imbue LLMs with System 2-like reasoning, though its evaluation is currently limited to mathematical tasks, suggesting future work to generalize to broader reasoning domains and open-ended content.

Abstract

Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0).

LLM2: Let Large Language Models Harness System 2 Reasoning

TL;DR

This work tackles the tendency of LLMs to produce undesirable outputs by leveraging a dual-process design that pairs a System 1 LLM with a System 2 process-based verifier. The verifier is trained with synthetic process-supervision data generated through a token quality exploration strategy, and it can steer generation at each token via a process score. Empirically, LLM2 improves mathematical reasoning on GSM8K and MATH across Llama3 and Qwen2.5 models, with self-consistency and self-generated data further boosting performance and demonstrating favorable latency compared to token-level decoding baselines. The approach offers a scalable way to imbue LLMs with System 2-like reasoning, though its evaluation is currently limited to mathematical tasks, suggesting future work to generalize to broader reasoning domains and open-ended content.

Abstract

Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs. We posit that these limitations are rooted in the foundational autoregressive architecture of LLMs, which inherently lacks mechanisms for differentiating between desirable and undesirable results. Drawing inspiration from the dual-process theory of human cognition, we introduce LLM2, a novel framework that combines an LLM (System 1) with a process-based verifier (System 2). Within LLM2, the LLM is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs. The verifier is trained with a pairwise comparison loss on synthetic process-supervision data generated through our token quality exploration strategy. Empirical results on mathematical reasoning benchmarks substantiate the efficacy of LLM2, exemplified by an accuracy enhancement from 50.3 to 57.8 (+7.5) for Llama3-1B on GSM8K. Furthermore, when combined with self-consistency, LLM2 achieves additional improvements, boosting major@20 accuracy from 56.2 to 70.2 (+14.0).
Paper Structure (35 sections, 6 equations, 3 figures, 10 tables)

This paper contains 35 sections, 6 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Dual-process LLM
  4. Process-based Verifier
  5. Synthetic Process-supervision
  6. Continuation Generation
  7. Quality Assessment
  8. Negative Sampling
  9. Experiments
  10. Experimental Setup
  11. Results
  12. Analysis
  13. Self-generated Answers for Synthetic Process-supervision
  14. Self-consistency
  15. Latency
  16. ...and 20 more sections

Figures (3)

  • Figure 1: An illustration of the training and inference stages of LLM2. The training stage includes (a) synthetic process-supervision data collection and (b) the optimization of a process-based verifier. The inference stage involves (c) a dual-process LLM for generation.
  • Figure 2: Results of LLM2 and other baselines' performance on GSM8K and MATH with Llama3 series.
  • Figure 3: Results on combining LLM2 with self-consistency on GSM8K and MATH using Llama3-1B.