<Research Paper>Validating Classroom Assessments Measuring Learner Knowledge of Academic Vocabulary

Abstract

This research investigates the reliability of vocabulary assessments used in English language skill classes at Kyoto University, Japan. Specifically it examines assessments designed to measure learner knowledge of a set of 477 vocabulary items from the Kyoto University Academic Vocabulary Database (京大学術語彙データベース : 基本英単語1110) in the first-year curriculum of a combined skills writing/listening course. Learners provided responses to two item types--multiple-choice and fill-in-the-blank--in an online quiz format constructed using Google Forms and delivered using QR codes. Results show that the vocabulary instruments revealed good model fit based on a Rasch analysis of each quiz version. Follow-up statistical analyses were performed in the form of t tests. These analyses, which used Rasch logits of item difficulty, revealed that items measuring receptive knowledge showed lower item difficulty than those measuring productive ability. A person-level analysis of gains over time showed somewhat mixed results, with just two of the five sample populations revealing gains between spring and fall semesters. This analysis was performed as part of a larger needs analysis project with results providing insights into how best to structure additional in-house created materials.