Title: The Method of Comparative Judgement for Assessment and Research

Authors: San Verhavert, Sven De Maeyer, Vincent Donche

Affiliation: University of Antwerp

Abstract:
Despite it being almost a century old, Thurstone's Law of Comparative Judgement
(1927) is still relevant today. It can be found under different forms in
psychophysics. It is, for example, the basis or the precursor of signal
detection theory (Macmillan & Creelman, 2005). The Law of Comparative Judgement
has also found its way (back) to education and educational assessment. The
method of Comparative Judgement (CJ) was first introduced as a research tool to
measure language proficiency (Pollitt & Murray, 1995) and more recently as a
method to assess complex competences and open-ended tasks in a more reliable and
valid way compared to (e.g.) rubrics scoring (Pollitt, 2004, 2009). An
assessment using CJ exists of a group of assessors each individually presented
with a series of pairs of student products (e.g. essays). For each pair the
assessor needs to indicate which of the two products is better regarding the
competence under assessment. Based on these judgements a ranking of product
quality is estimated. In this talk we will give an introduction to Comparative
Judgment as a method. We will mainly focus on the data structure and different
methods to estimate a ranking in R. More specifically we will use a function
that we implemented ourselves, the btm function from the sirt package
(Robitzsch, 2016), the BTm function from the BradleyTerry2 package (Turner &
Firth, 2012) and the vglm function in combination with the Brat and brat
functions from the VGAM package (Yee, 2010). We will also touch on several
(possible) applications of the CJ method both in education and elsewhere.