Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Text Scoring in the Age of Generative AI for the GPU-poor

Christopher Michael Ormerod, Alexander Kwako

TL;DR

This work investigates replacing large, proprietary GLMs with smaller open-source models to perform Automated Text Scoring (AES) and Automated Short Answer Scoring (ASAS) on GPU-poor hardware. It demonstrates memory-efficient fine-tuning using LoRA and 4-bit quantization (QLoRA) on four open-source models (Mistral, Gemma, Llama-3, Phi-3), achieving competitive AES/ASAS performance and enabling the generation of rubrics-based explanations for scores. While not achieving current state-of-the-art results on all benchmarks, the approach yields strong results relative to several baselines and enables transparent, researcher-controlled experimentation with model-generated feedback. The findings suggest a practical, cost-effective path for educational AI that supports explainable scoring while maintaining data security and reproducibility, with open-data release to foster further evaluation and development.

Abstract

Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.

Automated Text Scoring in the Age of Generative AI for the GPU-poor

TL;DR

This work investigates replacing large, proprietary GLMs with smaller open-source models to perform Automated Text Scoring (AES) and Automated Short Answer Scoring (ASAS) on GPU-poor hardware. It demonstrates memory-efficient fine-tuning using LoRA and 4-bit quantization (QLoRA) on four open-source models (Mistral, Gemma, Llama-3, Phi-3), achieving competitive AES/ASAS performance and enabling the generation of rubrics-based explanations for scores. While not achieving current state-of-the-art results on all benchmarks, the approach yields strong results relative to several baselines and enables transparent, researcher-controlled experimentation with model-generated feedback. The findings suggest a practical, cost-effective path for educational AI that supports explainable scoring while maintaining data security and reproducibility, with open-data release to foster further evaluation and development.

Abstract

Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.
Paper Structure (23 sections, 2 equations, 1 figure, 7 tables)

This paper contains 23 sections, 2 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Automated text scoring
  4. Model-generated feedback
  5. Architecture of Generative Language Models for the GPU Poor
  6. Training Generative Language Models for the GPU Poor
  7. Methods
  8. Data
  9. Performance Metric
  10. Models
  11. Parameter-efficient fine-tuning
  12. Prompting for Score Prediction
  13. Prompting for Feedback Generation
  14. Qualitative Analysis of Feedback
  15. Results
  16. ...and 8 more sections

Figures (1)

  • Figure 1: A visual representation of the training scheme for LoRA. The update of $W$ is given by $W_0 + BA$ where $B$ and $A$ are of a particular low-rank form.