Llama-Polya: Instruction Tuning for Large Language Model based on Polya's Problem-solving

TL;DR

The paper tackles the challenge of teaching mathematical problem-solving with pedagogy-aligned AI by embedding Polya's four-step framework into an instruction-tuned LLM, Llama-Polya. It builds a data pipeline that converts GSM8K problems into Polya-guided dialogues and fine-tunes Llama-3.1-8B with a full, non-PEFT approach using a ChatML format, evaluated across Arithmetic, Measurement, and Geometry. Results show that domain-specific, Polya-informed tuning yields more balanced reasoning stages and fewer premature answers, with expert evaluators noting improved pedagogical coherence and metacognitive prompting, though personalization and rigorous math checks remain areas for improvement. The work demonstrates that pedagogy-grounded instruction tuning can enhance educational alignment and reasoning transparency in AI tutoring, suggesting pathways for adaptive prompting and curriculum-aware fine-tuning in future research.

Abstract

This paper introduces Llama-Polya, an instruction-tuned large language model that integrates Polya's four-step problem-solving framework into its dialogue structure to support mathematical reasoning. Mathematical problem-solving is central to students' success in mathematics education, yet many learners struggle to plan, justify, and verify their solutions. Although large language models (LLMs) show promise as intelligent tutors, they often lack structured pedagogical alignment grounded in established learning theories. To address this gap, we operationalize Polya's problem-solving framework within an instruction-tuned LLM to promote metacognitive engagement and examine the effects of pedagogy-aligned fine-tuning compared to domain-only and general-purpose instruction tuning. Built on the Llama-3.1-8B architecture, Llama-Polya was fine-tuned on synthetic math problem-solving data derived from GSM8K, structured according to Polya's four stages. We developed and evaluated multiple variants-general-purpose instruct, math-domain metamath, pedagogy-aligned polya-v2, and sequential metamath+polya-v2-using both quantitative accuracy metrics and qualitative pedagogical assessments. Results indicate that models tuned with Polya's framework and domain-specific data produced more balanced reasoning-stage distributions and fewer premature answers. Expert evaluators also observed improved pedagogical coherence and metacognitive prompting, although limitations in personalization and mathematical rigor remained. These findings suggest that pedagogy-grounded instruction tuning can enhance educational alignment and reasoning transparency in LLM-based tutoring systems.

Figures (2)

• Figure 1: Process of train dataset generation. Each stage links to Polya's critical stages.
• Figure 2: Normalized Polya-stage distributions across models