Overview of AI Grading of Physics Olympiad Exams

TL;DR

The paper addresses the challenge of automatically grading diverse high school physics problems for the Australian Physics Olympiad. It conducts a December 2024 systematic literature review and proposes a multi-modal AI grading framework aligned with Australia's AI Ethics Principles. Findings indicate that numeric, algebraic, plots/diagrams, and short-answer items require different techniques—from rule-based and OCR approaches to LLMs, CAS, and multimodal models—each with trade-offs in accuracy and explainability. The authors advocate an LLM-modulo verification strategy to improve reliability and emphasize local, privacy-preserving deployment to reduce teacher workload while maintaining ethical guarantees.

Abstract

Automatically grading the diverse range of question types in high school physics problem is a challenge that requires automated grading techniques from different fields. We report the findings of a Systematic Literature Review of potential physics grading techniques. We propose a multi-modal AI grading framework to address these challenges and examine our framework in light of Australia's AI Ethical Principles.

Figures (5)

• Figure 1: Flow diagram illustrating the steps taken in the literature review and the number of papers included in each step. Details of the search terms and databases for each of the three sources can be found in Appendix \ref{['appendix:Search Terms']}.
• Figure 2: Details of search strategy used in the the Systematic Literature Review. Overall three searches were conducted, each with a different focus. Note that the databases include conference papers and books in addition to journal papers.
• Figure 3: Overlap of research papers found across different academic databases. The horizontal bar chart (left) shows the total number of papers from each source. Dark coloured bars indicate relevant papers while lighter coloured bars show the total number of papers retrieved. The upset plot (right) shows the overlap between each database.
• Figure 4: Histogram of the number of papers published for each automated grading technique and question type by year.
• Figure 5: Heatmap showing the distribution of different methods used across various question types in automated assessment systems. The colour intensity represents the logarithmic count of papers, while the numbers in each cell show the actual count. Empty cells indicate no papers were found for that combination.