Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training

TL;DR

The paper tackles the challenge of inducing robust medical reasoning in small Persian language models by combining Reinforcement Learning from AI Feedback (RLAIF) with Direct Preference Optimization (DPO) on a translated Persian MedMCQA dataset. Through a teacher–student setup and two data-generation strategies, the authors craft high-signal Chain-of-Thought preference data and train gaokerena-R, an 8B-parameter Persian model, to outperform a larger data-trained baseline on reasoning tasks. Results show gaokerena-R exhibits stronger multi-step reasoning under CoT prompting, while the larger pretraining corpus of gaokerena-V yields superior knowledge recall under direct prompting. The study highlights the value of reasoning-focused fine-tuning for low-resource, domain-specific NLP and discusses environmental benefits from using smaller, targeted datasets.

Abstract

Enhancing reasoning capabilities in small language models is critical for specialized applications such as medical question answering, particularly in underrepresented languages like Persian. In this study, we employ Reinforcement Learning with AI Feedback (RLAIF) and Direct preference optimization (DPO) to improve the reasoning skills of a general-purpose Persian language model. To achieve this, we translated a multiple-choice medical question-answering dataset into Persian and used RLAIF to generate rejected-preferred answer pairs, which are essential for DPO training. By prompting both teacher and student models to produce Chain-of-Thought (CoT) reasoning responses, we compiled a dataset containing correct and incorrect reasoning trajectories. This dataset, comprising 2 million tokens in preferred answers and 2.5 million tokens in rejected ones, was used to train a baseline model, significantly enhancing its medical reasoning capabilities in Persian. Remarkably, the resulting model outperformed its predecessor, gaokerena-V, which was trained on approximately 57 million tokens, despite leveraging a much smaller dataset. These results highlight the efficiency and effectiveness of reasoning-focused training approaches in developing domain-specific language models with limited data availability.

Figures (5)

• Figure 1: Method 1 Block Diagram
• Figure 2: Method 2 Block Diagram
• Figure 3: Training loss curve
• Figure 4: Pass@k results on the FA_MED_MMLU dataset using Chain-of-Thought prompting
• Figure 5: Pass@k results on the IBMSEE (September 2023) dataset using Chain-of-Thought prompting