Molly: Making Large Language Model Agents Solve Python Problem More Logically

TL;DR

This work addresses the challenge of making large language models act as effective programming teaching assistants for Chinese Python learners. It introduces Molly, a three-stage LLM agent that combines scenario-based intent detection, knowledge retrieval from a structured educational knowledge base, and iterative self-reflection to improve answer accuracy, expressiveness, and usefulness. A Chinese Python QA dataset (5,960 QA pairs, built from 16,247 questions) with expert annotations underpins the knowledge base, enabling pedagogy-aligned responses. Across multiple LLMs, Molly improves teaching-oriented outcomes, demonstrating potential for scalable, dialog-based programming education.

Abstract

Applying large language models (LLMs) as teaching assists has attracted much attention as an integral part of intelligent education, particularly in computing courses. To reduce the gap between the LLMs and the computer programming education expert, fine-tuning and retrieval augmented generation (RAG) are the two mainstream methods in existing researches. However, fine-tuning for specific tasks is resource-intensive and may diminish the model`s generalization capabilities. RAG can perform well on reducing the illusion of LLMs, but the generation of irrelevant factual content during reasoning can cause significant confusion for learners. To address these problems, we introduce the Molly agent, focusing on solving the proposed problem encountered by learners when learning Python programming language. Our agent automatically parse the learners' questioning intent through a scenario-based interaction, enabling precise retrieval of relevant documents from the constructed knowledge base. At generation stage, the agent reflect on the generated responses to ensure that they not only align with factual content but also effectively answer the user's queries. Extensive experimentation on a constructed Chinese Python QA dataset shows the effectiveness of the Molly agent, indicating an enhancement in its performance for providing useful responses to Python questions.

Figures (5)

• Figure 1: Molly-Agent compared with two mainstream programming QA employing LLMs.
• Figure 2: The framework of MollyAgent. Molly operates in three stages: (1) Agent Perception. In this stage, we use large models to further clarify the user's questions and provide more relevant information for the learner's questions to improve retrieval accuracy. (2) Knowledge Retrieval and Answer Generation. At this stage, we combine the learner's questions with the LLM's understanding to retrieve relevant answers from the constructed structured knowledge base. We then use the LLM to generate appropriate answers based on the retrieved relevant answers. (3) In the final stage, we iteratively refine the answers generated in the previous phase using role-playing capabilities to ensure their accuracy and usefulness.
• Figure 3: The Prompt Template of Answer Reflection Stage. We illustrate Molly’s prompt template during the final stage. Through phased observation and evaluation, answers generated in the second stage are refined using the self-reflective capabilities of a large model. This further optimization improves the model's ability to respond to learners' questions.
• Figure 4: The effectiveness of the intention recognition based on role-playing and self-reflection.
• Figure 5: Comparative results of using different methods as knowledge bases.