Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Moving Beyond Review: Applying Language Models to Planning and Translation in Reflection

Seyed Parsa Neshaei, Richard Lee Davis, Tanja Käser

Abstract

Reflective writing is known to support the development of students' metacognitive skills, yet learners often struggle to engage in deep reflection, limiting learning gains. Although large language models (LLMs) have been shown to improve writing skills, their use as conversational agents for reflective writing has produced mixed results and has largely focused on providing feedback on reflective texts, rather than support during planning and organizing. In this paper, inspired by the Cognitive Process Theory of writing (CPT), we propose the first application of LLMs to the planning and translation steps of reflective writing. We introduce Pensée, a tool to explore the effects of explicit AI support during these stages by scaffolding structured reflection planning using a conversational agent, and supporting translation by automatically extracting key concepts. We evaluate Pensée in a controlled between-subjects experiment (N=93), manipulating AI support across writing phases. Results show significantly greater reflection depth and structural quality when learners receive support during planning and translation stages of CPT, though these effects reduce in a delayed post-test. Analyses of learner behavior and perceptions further illustrate how CPT-aligned conversational support shapes reflection processes and learner experience, contributing empirical evidence for theory-driven uses of LLMs in AI-supported reflective writing.

Moving Beyond Review: Applying Language Models to Planning and Translation in Reflection

Abstract

Reflective writing is known to support the development of students' metacognitive skills, yet learners often struggle to engage in deep reflection, limiting learning gains. Although large language models (LLMs) have been shown to improve writing skills, their use as conversational agents for reflective writing has produced mixed results and has largely focused on providing feedback on reflective texts, rather than support during planning and organizing. In this paper, inspired by the Cognitive Process Theory of writing (CPT), we propose the first application of LLMs to the planning and translation steps of reflective writing. We introduce Pensée, a tool to explore the effects of explicit AI support during these stages by scaffolding structured reflection planning using a conversational agent, and supporting translation by automatically extracting key concepts. We evaluate Pensée in a controlled between-subjects experiment (N=93), manipulating AI support across writing phases. Results show significantly greater reflection depth and structural quality when learners receive support during planning and translation stages of CPT, though these effects reduce in a delayed post-test. Analyses of learner behavior and perceptions further illustrate how CPT-aligned conversational support shapes reflection processes and learner experience, contributing empirical evidence for theory-driven uses of LLMs in AI-supported reflective writing.

Paper Structure

This paper contains 13 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Pensée: Reflective Learning with CPT
  3. User Interface Design
  4. AI Models
  5. Experimental User Study
  6. Measures and Analysis
  7. Results
  8. RQ1: Effects of Explicit CPT-oriented Support Across Stages
  9. RQ2: Response Patterns and Interaction Behavior
  10. RQ3: User Experience and Perception Metrics
  11. Discussion

Figures (4)

  • Figure 1: Interface of Pensée consisting of two screens. Left: A CA guides learners in the planning phase, with key concepts automatically being extracted to support translation. Right: Writing area where learners compose their text using key concepts and receive automated feedback on the text structure during the review stage.
  • Figure 2: Left: the reflection cycle from Glogger et al. glogger2012learning, starting from top-left, used in designing and steering the model behind our CA. Right: the architecture for the different LLM agents used in the planning and translation stages in our tool.
  • Figure 3: Comparing the depth score of reflections across different stages and groups.
  • Figure 4: Self-reported perception metrics by group (Mean $\pm$ SEM).