Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

Sahar Yarmohammadtoosky, Yiyun Zhou, Victoria Yaneva, Peter Baldwin, Saed Rezayi, Brian Clauser, Polina Harikeo

TL;DR

This work probes the robustness of transformer-based automated short-answer grading in medical education against adversarial gaming, proposing adversarial training, ensemble defenses, and GPT-4 prompting as countermeasures. Across dataset- and model-based experiments, adversarial training with diverse gaming strategies and ensemble methods substantially reduces false positives while preserving high accuracy on real responses; GPT-4 prompting further enhances detection of gaming efforts. The findings demonstrate actionable improvements in robustness and reliability of AI-driven ASAG tools, with implications for fairness and trust in high-stakes educational assessments. They also highlight ongoing challenges, particularly for sophisticated or unseen adversarial strategies, and underscore the need for careful ethical considerations and continuous evaluation in deployment.

Abstract

This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the system's weaknesses, potentially leading to false positives. To counteract these vulnerabilities, we implement several adversarial training methods designed to enhance the systems' robustness. Our results indicate that these methods significantly reduce the susceptibility of grading systems to such manipulations, especially when combined with ensemble techniques like majority voting and ridge regression, which further improve the system's defense against sophisticated adversarial inputs. Additionally, employing large language models such as GPT-4 with varied prompting techniques has shown promise in recognizing and scoring gaming strategies effectively. The findings underscore the importance of continuous improvements in AI-driven educational tools to ensure their reliability and fairness in high-stakes settings.

Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems

TL;DR

This work probes the robustness of transformer-based automated short-answer grading in medical education against adversarial gaming, proposing adversarial training, ensemble defenses, and GPT-4 prompting as countermeasures. Across dataset- and model-based experiments, adversarial training with diverse gaming strategies and ensemble methods substantially reduces false positives while preserving high accuracy on real responses; GPT-4 prompting further enhances detection of gaming efforts. The findings demonstrate actionable improvements in robustness and reliability of AI-driven ASAG tools, with implications for fairness and trust in high-stakes educational assessments. They also highlight ongoing challenges, particularly for sophisticated or unseen adversarial strategies, and underscore the need for careful ethical considerations and continuous evaluation in deployment.

Abstract

This study examines vulnerabilities in transformer-based automated short-answer grading systems used in medical education, with a focus on how these systems can be manipulated through adversarial gaming strategies. Our research identifies three main types of gaming strategies that exploit the system's weaknesses, potentially leading to false positives. To counteract these vulnerabilities, we implement several adversarial training methods designed to enhance the systems' robustness. Our results indicate that these methods significantly reduce the susceptibility of grading systems to such manipulations, especially when combined with ensemble techniques like majority voting and ridge regression, which further improve the system's defense against sophisticated adversarial inputs. Additionally, employing large language models such as GPT-4 with varied prompting techniques has shown promise in recognizing and scoring gaming strategies effectively. The findings underscore the importance of continuous improvements in AI-driven educational tools to ensure their reliability and fairness in high-stakes settings.
Paper Structure (19 sections, 6 figures, 5 tables)

This paper contains 19 sections, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset
  5. ACTA System
  6. Gaming Strategies Method
  7. Prompt Engineering Method
  8. Experiments and Results
  9. ACTA Performance with Gaming Data
  10. Adversarial Training Experiments
  11. Post Adversarial Training Results
  12. Prompt Engineering Experiments
  13. Post Prompt Engineering Results
  14. Error Analysis
  15. Error Analysis For Adversarial Training
  16. ...and 4 more sections

Figures (6)

  • Figure 1: PCA of Response Embedding for Item 1
  • Figure 2: PCA of Response Embedding for Item 2
  • Figure 3: Adversarial Defense Workflow. Gaming data is combined with real data for training and testing purposes.
  • Figure 4: FPR Across Different Models - Part 1
  • Figure 5: FPR Across Different Models - Part 2
  • ...and 1 more figures