Do LLMs Need Inherent Reasoning Before Reinforcement Learning? A Study in Korean Self-Correction

TL;DR

This study addresses the challenge of boosting Korean reasoning in LLMs by examining whether reinforcement learning can match English-level improvements. It finds that RL alone yields limited gains unless the model already possesses strong Korean reasoning, due in part to early-layer internal translation of Korean inputs into English. The authors show that fine-tuning language-specific neurons in early layers, guided by a self-correction code-switching dataset, effectively aligns internal English-based reasoning with Korean inputs and yields substantial improvements in mathematical reasoning and self-correction. when combined with GRPO-based RL after this alignment, the approach delivers the most robust gains, highlighting a neuron-level alignment strategy for multilingual reasoning rather than merely injecting new language capabilities.

Abstract

Large Language Models (LLMs) demonstrate strong reasoning and self-correction abilities in high-resource languages like English, but their performance remains limited in low-resource languages such as Korean. In this study, we investigate whether reinforcement learning (RL) can enhance Korean reasoning abilities to a degree comparable to English. Our findings reveal that RL alone yields limited improvements when applied to models lacking inherent Korean reasoning capabilities. To address this, we explore several fine-tuning strategies and show that aligning the model's internal reasoning processes with Korean inputs-particularly by tuning Korean-specific neurons in early layers-is key to unlocking RL's effectiveness. We introduce a self-correction code-switching dataset to facilitate this alignment and observe significant performance gains in both mathematical reasoning and self-correction tasks. Ultimately, we conclude that the crucial factor in multilingual reasoning enhancement is not injecting new linguistic knowledge, but effectively eliciting and aligning existing reasoning capabilities. Our study provides a new perspective on how internal translation and neuron-level tuning contribute to multilingual reasoning alignment in LLMs.

Figures (9)

• Figure 1: Following zhaolarge, we decode the hidden embeddings of an LLM (Llama3.1-8B used in this example) into the vocabulary space. In the early layers, the model internally translates Korean inputs into English. We also observe that the LLM struggles to perform this internal translation for inputs related to mathematical reasoning and self-correction.
• Figure 2: Overall Process of Self-Correction Dataset Construction.
• Figure 3: Layer-wise comparison of hidden representations for English vs. Chinese and Korean inputs on general QA and MATH datasets.
• Figure 4: Ratio of activated Korean-specific neurons across tasks (Llama3.1-8B model).
• Figure 5: Results of self-correction on Korean GSM8K and MATH datasets across various models after applying GRPO. Llama3.1-8B is used for this experiment. Successful self-correction (Y-axis) refers to instances where the model exhibits self-correcting behavior that ultimately leads to the correct answer.