Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education

TL;DR

This paper presents TeachYou, an LBT system where learners teach an LLM-based tutee (AlgoBo) to code algorithms. It introduces the Reflect-Respond prompting pipeline to confine and update AlgoBo's knowledge, Mode-shifting to induce elaboration, and Teaching Helper to provide metacognitive feedback. Technical evaluation shows that Reflect-Respond can configure, persist, and adapt AlgoBo's knowledge state, while a 40-person user study demonstrates that Mode-shifting significantly increases knowledge-building density (effect size $d=0.71$, $p=0.03$) and enhances perceived usefulness for learning. The work argues for cost-efficient, personalized, and scalable LBT with teachable agents, and discusses implications for design, personalization, and classroom deployment.

Abstract

This work investigates large language models (LLMs) as teachable agents for learning by teaching (LBT). LBT with teachable agents helps learners identify knowledge gaps and discover new knowledge. However, teachable agents require expensive programming of subject-specific knowledge. While LLMs as teachable agents can reduce the cost, LLMs' expansive knowledge as tutees discourages learners from teaching. We propose a prompting pipeline that restrains LLMs' knowledge and makes them initiate "why" and "how" questions for effective knowledge-building. We combined these techniques into TeachYou, an LBT environment for algorithm learning, and AlgoBo, an LLM-based tutee chatbot that can simulate misconceptions and unawareness prescribed in its knowledge state. Our technical evaluation confirmed that our prompting pipeline can effectively configure AlgoBo's problem-solving performance. Through a between-subject study with 40 algorithm novices, we also observed that AlgoBo's questions led to knowledge-dense conversations (effect size=0.71). Lastly, we discuss design implications, cost-efficiency, and personalization of LLM-based teachable agents.

Figures (5)

• Figure 1: To the left, the 3 learning objectives they need to reach (A), learners can see AlgoBo's profile (B), and the questions they need to help AlgoBo solve (C). To the right, they can see the code they submitted (E), a code playground (F), and the code that AlgoBo write (G). When AlgoBo wrote code, participants could click on the "run test cases" and run AlgoBo's code. In the middle (D), learners use a typical chat interface to teach AlgoBo while receiving questions (H) and guidance from Teaching Helper (I)
• Figure 2: The overview of the Reflect-Respond prompting pipeline for simulating knowledge learning of AlgoBo and examples for each component. From the recent conversation, AlgoBo extracts new knowledge of the while loop condition and update its knowledge state (colored in green). Then, AlgoBo retrieves knowledge relevant to while loops and composes a response that fills its knowledge gap.
• Figure 3: AlgoBo shifts its mode in every three messages. When AlgoBo is in the questioner mode, it keeps asking follow-up questions until receiving a satisfactory response (constructive loop)
• Figure 4: The four Teaching Helper messages and corresponding suggestions that appear depending on the conversational patterns.
• Figure 5: The process of measuring adaptability for correct tutoring with an Implementation problem and State 2 as a seed knowledge state. The evaluations were performed in Korean to ensure compatibility with the main study conditions.