Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction

Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Hyunseung Lim, Yoonsu Kim, Tak Yeon Lee, Hwajung Hong, Juho Kim, So-Yeon Ahn, Alice Oh

TL;DR

This work explores how LLMs can function as English tutors, providing effective essay feedback tailored to students, and proposes three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects.

Abstract

In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding quality and characteristics. On the other hand, EFL learners assess their learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.

LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction

TL;DR

This work explores how LLMs can function as English tutors, providing effective essay feedback tailored to students, and proposes three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects.

Abstract

In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three metrics to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding quality and characteristics. On the other hand, EFL learners assess their learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.
Paper Structure (25 sections, 2 figures, 6 tables)

This paper contains 25 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. LLMs as EFL Tutors: Early Insights
  3. Advantage of LLM-as-a-tutor
  4. Limitation of LLM-as-a-tutor
  5. Mitigating Limitation
  6. Student-LLM Interaction Evaluation
  7. Annotator Details
  8. Evaluation Details
  9. Targets
  10. Perspectives
  11. Criteria
  12. Quality
  13. Characteristics
  14. Learning Outcome
  15. Results
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Evaluation results on quality and characteristic of two rubric-based feedback with standard prompting and score-based prompting in a 7-point Likert scale. C, O, and L denote Content, Organization, and Language, respectively. Asterisk denotes statistical significance tested by the paired T-test at $p$ level of $< 0.05$.
  • Figure 2: Learning outcome