Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions

Zhengyuan Liu, Stella Xin Yin, Carolyn Lee, Nancy F. Chen

TL;DR

This work investigates scaffolding in intelligent tutoring systems for language learning using multi-modal LLM-powered tutors. It develops a case study where GPT-4V guides children to describe images, grounding interactions in four learning theories and evaluating scaffolding with a seven-dimension rubric. The studyFurther introduces automated scaffolding evaluation via in-context learning in LLMs, enabling scalable benchmarking of conversational tutors. Findings indicate LLM-based tutors can follow pedagogical instructions and support self-paced learning across student groups, with implications for scalable, personalized language education.

Abstract

Intelligent tutoring systems (ITSs) that imitate human tutors and aim to provide immediate and customized instructions or feedback to learners have shown their effectiveness in education. With the emergence of generative artificial intelligence, large language models (LLMs) further entitle the systems to complex and coherent conversational interactions. These systems would be of great help in language education as it involves developing skills in communication, which, however, drew relatively less attention. Additionally, due to the complicated cognitive development at younger ages, more endeavors are needed for practical uses. Scaffolding refers to a teaching technique where teachers provide support and guidance to students for learning and developing new concepts or skills. It is an effective way to support diverse learning needs, goals, processes, and outcomes. In this work, we investigate how pedagogical instructions facilitate the scaffolding in ITSs, by conducting a case study on guiding children to describe images for language learning. We construct different types of scaffolding tutoring systems grounded in four fundamental learning theories: knowledge construction, inquiry-based learning, dialogic teaching, and zone of proximal development. For qualitative and quantitative analyses, we build and refine a seven-dimension rubric to evaluate the scaffolding process. In our experiment on GPT-4V, we observe that LLMs demonstrate strong potential to follow pedagogical instructions and achieve self-paced learning in different student groups. Moreover, we extend our evaluation framework from a manual to an automated approach, paving the way to benchmark various conversational tutoring systems.

Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions

TL;DR

This work investigates scaffolding in intelligent tutoring systems for language learning using multi-modal LLM-powered tutors. It develops a case study where GPT-4V guides children to describe images, grounding interactions in four learning theories and evaluating scaffolding with a seven-dimension rubric. The studyFurther introduces automated scaffolding evaluation via in-context learning in LLMs, enabling scalable benchmarking of conversational tutors. Findings indicate LLM-based tutors can follow pedagogical instructions and support self-paced learning across student groups, with implications for scalable, personalized language education.

Abstract

Intelligent tutoring systems (ITSs) that imitate human tutors and aim to provide immediate and customized instructions or feedback to learners have shown their effectiveness in education. With the emergence of generative artificial intelligence, large language models (LLMs) further entitle the systems to complex and coherent conversational interactions. These systems would be of great help in language education as it involves developing skills in communication, which, however, drew relatively less attention. Additionally, due to the complicated cognitive development at younger ages, more endeavors are needed for practical uses. Scaffolding refers to a teaching technique where teachers provide support and guidance to students for learning and developing new concepts or skills. It is an effective way to support diverse learning needs, goals, processes, and outcomes. In this work, we investigate how pedagogical instructions facilitate the scaffolding in ITSs, by conducting a case study on guiding children to describe images for language learning. We construct different types of scaffolding tutoring systems grounded in four fundamental learning theories: knowledge construction, inquiry-based learning, dialogic teaching, and zone of proximal development. For qualitative and quantitative analyses, we build and refine a seven-dimension rubric to evaluate the scaffolding process. In our experiment on GPT-4V, we observe that LLMs demonstrate strong potential to follow pedagogical instructions and achieve self-paced learning in different student groups. Moreover, we extend our evaluation framework from a manual to an automated approach, paving the way to benchmark various conversational tutoring systems.
Paper Structure (19 sections, 3 figures, 3 tables)

This paper contains 19 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Intelligent Tutoring Systems
  4. Scaffolding in Children’s Language Learning
  5. Multi-modal Tutoring Systems with Pedagogical Instructions
  6. Tutoring Language Learning via Image Description
  7. Multi-modal Systems as an Image Description Tutor
  8. Enhancing Tutoring Systems with Pedagogical Instructions
  9. Evaluating Tutoring Systems from Scaffolding Perspective
  10. Building Student Capability Levels for Evaluation
  11. Building Scaffolding Evaluation Rubric
  12. Experimental Setting and Qualitative Analysis
  13. Experimental Results and Analysis
  14. Comparison between high- and low-ability students
  15. Comparison among Scaffolding Tutors
  16. ...and 4 more sections

Figures (3)

  • Figure 1: A dialogue example of interactive language learning via an image description tutoring system. The student is asked to describe the picture.
  • Figure 2: Coding result of the five systems with different pedagogical instructions (Up: high-ability group; Down: low-ability group).
  • Figure 3: Normalized capability scoring in seven dimensions of the five systems with different pedagogical instructions.