An Analysis of Rater Severity and Leniency in Open-ended Mathematic Questions Rated Through Standard Rubrics and Rubrics Based on the SOLO Taxonomy

Abstract

This study uses the many-facet Rasch model to analyze the severity and leniency of raters of open-ended mathematic questions rated through standard rubrics and rubrics based on the SOLO taxonomy. The data source of the study was obtained from 104 eight grade students based on their responses to open-ended questions on mathematics-achievement test that was created by the researchers. The study’s participants were seven mathematics teachers who serving as raters in the research. The data collection instruments involved standard rubrics and rubrics based on the SOLO taxonomy. The collection of the data was performed in a few phases. In the first phase, the mathematics achievement test, which included open-ended questions, was administered to the students and evaluated by the raters. Then, raters scored students' responses to the open-ended questions using standard rubrics. Next, the raters scored the responses using rubrics based on the SOLO taxonomy. The data thus acquired were analyzed using the many-facet Rasch model. The study found that the raters' agreement was low, and that there were significant differences between raters in terms of severity and leniency when they used standard rubrics. When the ratings were done with rubrics based on SOLO taxonomy, the raters were consistent and their scoring severity and leniency were similar to each other.