Giving students fair, justified and consistent feedback on their work is a cornerstone of good teaching practice. So how consistent are humans when they are grading programming assignments? Join us on Monday 3rd November at 2pm GMT (UTC) to discuss a paper [1], abstract below. :
Providing consistent summative assessment to students is important, as the grades they are awarded affect their progression through university and future career prospects. While small cohorts are typically assessed by a single assessor, such as the module/class leader, larger cohorts are often assessed by multiple assessors, typically teaching assistants, which increases the risk of inconsistent grading.
To investigate the consistency of human grading of programming assignments, we asked 28 participants to each grade 40 CS1 introductory Java assignments, providing grades and feedback for correctness, code elegance, readability and documentation; the 40 assignments were split into two batches of 20. The 28 participants were divided into seven groups of four (where each group graded the same 40 assignments) to allow us to investigate the consistency of a group of assessors. In the second batch of 20, we duplicated one assignment from the first to analyse the internal consistency of individual assessors.
We measured the inter-rater reliability of the groups using Krippendorff’s α—an α > 0.667 is recommended to make tentative conclusions based on the rating. Our groups were inconsistent, with an average α = 0.02 when grading correctness and an average α < 0.01 for code elegance, readability and documentation.
To measure the individual consistency of graders, we measured the distance between the grades they awarded for the duplicated assignment in batch one and batch two. Only one participant of the 22 who didn’t notice that the assignment was a duplicate was awarded the same grade for correctness, code elegance, readability and documentation. The average grade difference was 1.79 for correctness and less than 1.6 for code elegance, readability and documentation.
Our results show that human graders in our study cannot agree on the grade to give a piece of student work and are often individually inconsistent, suggesting that the idea of a ‘gold standard’ of human grading might be flawed. This highlights that a shared rubric alone is not enough to ensure consistency, and other aspects such as assessor training and alternative grading practices should be explored to improve the consistency of human grading further when grading programming assignments
We’ll be joined by the papers lead author Marcus Messer from King’s College London, who’ll give us a lightning talk summary of the research. All welcome, meeting URL is public at zoom.us/j/96465296256 (meeting ID 9646-5296-256) but the password is private and pinned in the slack channel which you can join by following the instructions at sigcse.cs.manchester.ac.uk/join-us
(Cite this article using DOI:10.59350/01cd6-dvq50 provided by rogue-scholar.org)
References
- Marcus Messer, Neil C. C. Brown, Michael Kölling, Miaojing Shi (2025) How Consistent Are Humans When Grading Programming Assignments? ACM Transactions on Computing Education, Volume 25, Issue 4 Article No.: 49, Pages 1 – 37 DOI:10.1145/3759256
- Neil Brown (2025) Consistency of grading programming assignments academiccomputing.wordpress.com
