Lessons Learned from Designing an Open-Source Automated Feedback System for STEM Education

TL;DR

RATsApp, an open-source automated feedback system (AFS) that incorporates research-based features such as formative feedback, is presented, confirming the applicability of the TAM2 framework and finding a linear relation between the perceived usefulness and the intention to use.

Abstract

As distance learning becomes increasingly important and artificial intelligence tools continue to advance, automated systems for individual learning have attracted significant attention. However, the scarcity of open-source online tools that are capable of providing personalized feedback has restricted the widespread implementation of research-based feedback systems. In this work, we present RATsApp, an open-source automated feedback system (AFS) that incorporates research-based features such as formative feedback. The system focuses on core STEM competencies such as mathematical competence, representational competence, and data literacy. It also allows lecturers to monitor students' progress. We conducted a survey based on the technology acceptance model (TAM2) among a set of students (N=64). Our findings confirm the applicability of the TAM2 framework, revealing that factors such as the relevance of the studies, output quality, and ease of use significantly influence the perceived usefulness. We also found a linear relation between the perceived usefulness and the intention to use, which in turn is a significant predictor of the frequency of use. Moreover, the formative feedback feature of RATsApp received positive feedback, indicating its potential as an educational tool. Furthermore, as an open-source platform, RATsApp encourages public contributions to its ongoing development, fostering a collaborative approach to improve educational tools.

Figures (6)

• Figure 1: RATsApp supports four distinct user types: Student, RAT Creator, Lecturer, and Administrator. The architecture of RATsApp is divided into three layers: the Presentation Layer, the System Layer, and the Database Layer. Each layer serves a specific function in the overall operation of the application.
• Figure 2: The student workflow of answering RATs: First a lecture has to be selected, afterwards a student can choose either one of answering automatically generated RATs, a RAT sheet, or a live RAT sheet. This will lead to a new page with the choice to answer the RAT, view scaffolds or statistics. After answering a RAT results and feedback are shown and the student is either guided to the next RAT in the sheet or shown the now updated competence levels, if the sheet is finished.
• Figure 3: Frequency of all participants which stated that they use the system more often than "never" and who were therefore categorized as users. The categories have been translated from German to English and their selection frequencies are relative to the number of users.
• Figure 4: Answer frequencies of use of participants who stated that they use the system more often than "never" and were therefore categorized as users. The use frequency categories (see Table \ref{['tab:used_questionnaire']} for more details), translated from German to English, show the selection frequencies relative to the number of users. The questions belonging to each item can be found in the Appendix \ref{['secA1']}.
• Figure 5: Distribution of selected rating for a category of the questionnaire relative to the number of all participants. The violin plot on the left in each category shows the rating of users of the platform, while the violin plot on the right shows the ratings of non-users. The different numbers in the figure correspond to a scale from 1="strongly disagree" to 7="strongly agree" (positive for the platform), with 4="neutral". The plot also includes a representation of participants who did not provide an answer. The categories were translated from German to English.