Evaluation of Systems Programming Exercises through Tailored Static Analysis

TL;DR

An experience report on static analysis for the automatic evaluation of systems programming exercises is presented and it is shown that static analysis can identify a significant number of erroneous submissions missed by test cases.

Abstract

In large programming classes, it takes a significant effort from teachers to evaluate exercises and provide detailed feedback. In systems programming, test cases are not sufficient to assess exercises, since concurrency and resource management bugs are difficult to reproduce. This paper presents an experience report on static analysis for the automatic evaluation of systems programming exercises. We design systems programming assignments with static analysis rules that are tailored for each assignment, to provide detailed and accurate feedback. Our evaluation shows that static analysis can identify a significant number of erroneous submissions missed by test cases.

Figures (4)

• Figure 1: Overview of the code evaluation process.
• Figure 2: Assignment on producer-consumer.
• Figure 3: Output of the producer-consumer program.
• Figure 4: Detection of commits with errors, by type of evaluation check.