レポート評価におけるルーブリックの開発とその信頼性の検討

Abstract

Performance assessments such as writing assessment have been generally thought to have low reliability, whereas they have high content and face validity. The purpose of this study is to develop a rubric for writing assessment and to examine the reliability of assessment with the rubric. In this study, we highlighted a writing assessment for the course “Study Skills at University” for the first-year students at the Faculty of Dentistry, Niigata University. This course was provided in the form of both lectures and seminars in the first semester. The outcomes of learning were assessed with student essays and presentations. We developed “Cross-Model Rubric, ” which comprises six dimensions with three levels, to articulate logical thinking, problem solving and communication as the desired outcomes of this course. One of us has given a class of academic writing, using the rubric developed in this study as one of teaching materials, and three raters including the lecturer assessed student essays (N ＝ 64). With the goal of assessing the reliability, we used generalizability theory to estimate the contributions of student, rater, tasks, and their interactions to total variance. The generalizability coefficient, which is analogous to the reliability coefficient in classical test theory, scored 0.62, which was acceptable but did not reach a desired value. The generalizability theory analysis found that the writing assessment should be administered by more raters to achieve sufficient reliability by distinguishing individual students. In terms of feasibility of assessment, however, we propose that the rubric should have more levels of dimensions and the descriptors should be modified to appropriately assess student achievement.