BALEAP 2023 conference

Weir (2005: 56) noted that “The last decade of the twentieth century saw a general decline in the prestige of psychometric, statistically-driven approaches to testing.” This has resulted in a move away from the use if discrete skill tests of reading, listening, writing, and speaking to measure student’s competence in academic skills, to assessments that are more holistic, authentic and integrated in nature. These types of assessments are often task-based, performance-based, or scenario-based, and for students there is now likely to be a greater emphasis on writing essays, case studies, reports, projects, research papers, and portfolios, and on giving seminars and presentations. One of the criticisms often directed towards this type of performance-based assessment is that, because of the greater emphasis on subjective human raters, the reliability of the assessment could be compromised. Indeed, there is much evidence to support the claim that human raters are often fickle, erratic, inconsistent, wayward, unpredictable, and bias (Lumley & McNamara, 1995).

The assessments we give to our students have a significant impact on their lives, ranging from their course grade, their success or failure on a course, their GPA, to whether or not they gain entry into a particular institution, whether or not they graduate, or whether or not they are permitted to enter a chosen country. When the stakes of an assessment are this high, it is imperative that we get the rating of written and spoken work as accurate and reliable as possible. Therefore, there is a critical and pressing need to mitigate against rater bias and ensure reliable rating by human raters. We need to deconstruct the processes, procedures, and practices we currently employ to rate students' performance, and reconstruct them to ensure bias free rating that is consistent and reliable.

In this presentation we will examine the different types of bias rating behaviour that raters can exhibit, including: leniency bias; harshness bias; central tendency bias; halo effect bias; horns effect bias; contrast effect bias; first impression bias; recency bias: similar-to-me bias; and dissimilar-to-me bias. Participants will have the opportunity to reflect on the types of rater bias that they may influence them when they mark their students written or spoken assessments. We will look at the causes of rater bias (Eckes, 2012), such as not complying with the rating criteria, the subjective interpretation of the rating criteria, biased use of rating criteria, disagreement with content, or offensive content. We will then move on to discuss the importance of consistency and reliability when marking writing and speaking, and we will explore how FACETS (Many-Faceted Rasch Measurement) (Linacre, 2017) can be used to identify rater inconsistency, and the extent to which a rater rates too harshly or too leniently. We will then consider what we can do to reduce rater bias in order to improve the reliability of our human-rated assessments. In the final part of the presentation, we will briefly explore the potential use of automated essay scoring as a means of reducing rater bias.