Data Science Students Perspectives on Learning Analytics: An Application of Human-Led and LLM Content Analysis
Raghda Zahran, Jianfei Xu, Huizhi Liang, Matthew Forshaw
TL;DR
This work addresses how students perceive and articulate learning analytics by studying postgraduate data science students using a mixed-method QCALA framework that combines human analysis with Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) on the anonymised Open University Learning Analytics Dataset (OULAD). The approach leverages CRISP-DM-guided data preparation, Exploratory Data Analysis, and multi-channel analysis to extract actionable insights from student reports. The findings show that data science specialists articulate nuanced analytics questions and that LLMs provide deeper contextual interpretations, suggesting complementary strengths of human and AI analyses. The study demonstrates the value of a human-centered, participatory design approach for learning analytics and offers a methodological blueprint for integrating qualitative and AI-assisted analyses to inform analytics design in higher education.
Abstract
Objective This study is part of a series of initiatives at a UK university designed to cultivate a deep understanding of students' perspectives on analytics that resonate with their unique learning needs. It explores collaborative data processing undertaken by postgraduate students who examined an Open University Learning Analytics Dataset (OULAD). Methods A qualitative approach was adopted, integrating a Retrieval-Augmented Generation (RAG) and a Large Language Model (LLM) technique with human-led content analysis to gather information about students' perspectives based on their submitted work. The study involved 72 postgraduate students in 12 groups. Findings The analysis of group work revealed diverse insights into essential learning analytics from the students' perspectives. All groups adopted a structured data science methodology. The questions formulated by the groups were categorised into seven themes, reflecting their specific areas of interest. While there was variation in the selected variables to interpret correlations, a consensus was found regarding the general results. Conclusion A significant outcome of this study is that students specialising in data science exhibited a deeper understanding of learning analytics, effectively articulating their interests through inferences drawn from their analyses. While human-led content analysis provided a general understanding of students' perspectives, the LLM offered nuanced insights.