A study of approaches to assessment by portfolio for general practice trainers
Judging the quality of reflective portfolios is assuming critical importance
with their increasing use in accreditation and revalidation. The experience of this
work has shown that despite explicit instructions to compilers, considerable
investment in assessor training, and the negotiation, agreement and publication of
overt criteria, individual assessments are consistent but show only fair inter-rater
reliability and are untrustworthy in high stakes assessment. This study has also
shown that reliability is not improved through the process of trying to 'shape'
portfolios to introduce structural consistency, although it can be improved by using
discussant pairs of assessors, with consequent issues of cost.
While this method should be further researched as a means by which
portfolios can be judged, consideration of the nature of professional practice and the
processes that underpin judgements made by professionals moves the debate
towards rejecting the traditional first hurdle of reliability before examining the
validity of assessment tools. This thesis describes a methodology that begins the
process by justifying the general direction such enquiries might be taking. Based on
an exploration of the issues, it offers a model for a system for the assessment of
professionals that is both theoretically supported and practically possible.