And the winner is …: inter-rater reliability among scholarship assessors

With increasing competition for postgraduate research scholarships, awarding processes demand attention and scrutiny. We examine inter-rater reliability for two prestigious New Zealand scholarships, the Shirtcliffe Fellowship and the Gordon Watson Scholarship. For each scholarship, five assessors (three academic; two non-academic) independently evaluate all applicants over three domains: Academic Merit, Quality of Study Plans and Character/Leadership. Data from years 2009 to 2014 were extracted, comprising 12 separate assessment rounds. Good to excellent agreement was observed for each scholarship in each year. Agreement was significantly higher for the Academic Merit domain compared to the other domains. Moreover, agreement among academics was higher and less variable than non-academics for this Academic Merit domain. No such differences were noted in the other domains. While resource efficiencies could be made, reductions in committee size resulted in poorer applicant selection performance. Applicants and donors alike can be confident that the awardee for these scholarships is a top applicant.