Exploring subjectivity in competency-based assessment judgements of assessors

Rachel Bacon, Claire Palermo

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Aim: The aim of this research was to measure variations in assessors' judgements of a student dietitian's performance and to explore the influence of group discussion on their judgements. Methods: The assessments of a student's performance, as observed from a video recording of an authentic nutrition consultation, were measured pre- and post-group discussion by 26 experienced assessors using a mixed-methods questionnaire. The instrument included a validated 7-point visual analogue scale (VAS) rating (1=novice; 7=competent), a qualitative global description of performance and an assessor's confidence rating (1=not at all confident; 10=extremely confident). Scales were analysed descriptively and qualitative responses coded for key themes. Results: No agreement was found in assessors' rating in either the pretest (median=4, range=5) or post-test (median=4, range=4); however, the discussion led 78% of participants (20/26) to change their VAS ratings (9/26) and/or confidence levels (16/26). Three themes emerged from the thematic analysis of the participants' global descriptions of performance: (i) discourse supports assessors to justify their judgements, identify assumptions and learn from the observations of others; (ii) discourse leads assessors to more holistic judgements; and (iii) multiple sources of evidence and student reflections are necessary for credible judgement. Conclusions: This research questions the notion that 'actual' performance can be objectively measured and, rather, considers assessments as 'interpretations'. This research calls for an integrated interpretivist student-centred approach to competency-based assessment.

Original languageEnglish
Pages (from-to)357-364
Number of pages8
JournalNutrition and Dietetics
Volume74
Issue number4
DOIs
Publication statusPublished - 2017

Fingerprint

Students
Visual Analog Scale
Research
Video Recording
Nutritionists
Referral and Consultation
Surveys and Questionnaires

Cite this

@article{3d3484abcedc486f88c44d8841c9dc17,
title = "Exploring subjectivity in competency-based assessment judgements of assessors",
abstract = "Aim: The aim of this research was to measure variations in assessors' judgements of a student dietitian's performance and to explore the influence of group discussion on their judgements. Methods: The assessments of a student's performance, as observed from a video recording of an authentic nutrition consultation, were measured pre- and post-group discussion by 26 experienced assessors using a mixed-methods questionnaire. The instrument included a validated 7-point visual analogue scale (VAS) rating (1=novice; 7=competent), a qualitative global description of performance and an assessor's confidence rating (1=not at all confident; 10=extremely confident). Scales were analysed descriptively and qualitative responses coded for key themes. Results: No agreement was found in assessors' rating in either the pretest (median=4, range=5) or post-test (median=4, range=4); however, the discussion led 78{\%} of participants (20/26) to change their VAS ratings (9/26) and/or confidence levels (16/26). Three themes emerged from the thematic analysis of the participants' global descriptions of performance: (i) discourse supports assessors to justify their judgements, identify assumptions and learn from the observations of others; (ii) discourse leads assessors to more holistic judgements; and (iii) multiple sources of evidence and student reflections are necessary for credible judgement. Conclusions: This research questions the notion that 'actual' performance can be objectively measured and, rather, considers assessments as 'interpretations'. This research calls for an integrated interpretivist student-centred approach to competency-based assessment.",
keywords = "Assessment, Clinical nutrition and dietetics, Competency-based education, Professional competence, assessment, professional competence, clinical nutrition and dietetics, competency-based education",
author = "Rachel Bacon and Claire Palermo",
year = "2017",
doi = "10.1111/1747-0080.12326",
language = "English",
volume = "74",
pages = "357--364",
journal = "Nutrition and Dietetics",
issn = "1032-1322",
publisher = "Wiley-Blackwell",
number = "4",

}

Exploring subjectivity in competency-based assessment judgements of assessors. / Bacon, Rachel; Palermo, Claire.

In: Nutrition and Dietetics, Vol. 74, No. 4, 2017, p. 357-364.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Exploring subjectivity in competency-based assessment judgements of assessors

AU - Bacon, Rachel

AU - Palermo, Claire

PY - 2017

Y1 - 2017

N2 - Aim: The aim of this research was to measure variations in assessors' judgements of a student dietitian's performance and to explore the influence of group discussion on their judgements. Methods: The assessments of a student's performance, as observed from a video recording of an authentic nutrition consultation, were measured pre- and post-group discussion by 26 experienced assessors using a mixed-methods questionnaire. The instrument included a validated 7-point visual analogue scale (VAS) rating (1=novice; 7=competent), a qualitative global description of performance and an assessor's confidence rating (1=not at all confident; 10=extremely confident). Scales were analysed descriptively and qualitative responses coded for key themes. Results: No agreement was found in assessors' rating in either the pretest (median=4, range=5) or post-test (median=4, range=4); however, the discussion led 78% of participants (20/26) to change their VAS ratings (9/26) and/or confidence levels (16/26). Three themes emerged from the thematic analysis of the participants' global descriptions of performance: (i) discourse supports assessors to justify their judgements, identify assumptions and learn from the observations of others; (ii) discourse leads assessors to more holistic judgements; and (iii) multiple sources of evidence and student reflections are necessary for credible judgement. Conclusions: This research questions the notion that 'actual' performance can be objectively measured and, rather, considers assessments as 'interpretations'. This research calls for an integrated interpretivist student-centred approach to competency-based assessment.

AB - Aim: The aim of this research was to measure variations in assessors' judgements of a student dietitian's performance and to explore the influence of group discussion on their judgements. Methods: The assessments of a student's performance, as observed from a video recording of an authentic nutrition consultation, were measured pre- and post-group discussion by 26 experienced assessors using a mixed-methods questionnaire. The instrument included a validated 7-point visual analogue scale (VAS) rating (1=novice; 7=competent), a qualitative global description of performance and an assessor's confidence rating (1=not at all confident; 10=extremely confident). Scales were analysed descriptively and qualitative responses coded for key themes. Results: No agreement was found in assessors' rating in either the pretest (median=4, range=5) or post-test (median=4, range=4); however, the discussion led 78% of participants (20/26) to change their VAS ratings (9/26) and/or confidence levels (16/26). Three themes emerged from the thematic analysis of the participants' global descriptions of performance: (i) discourse supports assessors to justify their judgements, identify assumptions and learn from the observations of others; (ii) discourse leads assessors to more holistic judgements; and (iii) multiple sources of evidence and student reflections are necessary for credible judgement. Conclusions: This research questions the notion that 'actual' performance can be objectively measured and, rather, considers assessments as 'interpretations'. This research calls for an integrated interpretivist student-centred approach to competency-based assessment.

KW - Assessment

KW - Clinical nutrition and dietetics

KW - Competency-based education

KW - Professional competence

KW - assessment

KW - professional competence

KW - clinical nutrition and dietetics

KW - competency-based education

UR - http://www.scopus.com/inward/record.url?scp=85000444291&partnerID=8YFLogxK

UR - http://www.mendeley.com/research/exploring-subjectivity-competencybased-assessment-judgements-assessors

U2 - 10.1111/1747-0080.12326

DO - 10.1111/1747-0080.12326

M3 - Article

VL - 74

SP - 357

EP - 364

JO - Nutrition and Dietetics

JF - Nutrition and Dietetics

SN - 1032-1322

IS - 4

ER -