The Effects of a Rubric on Inexperienced Raters’ Scoring Consistency

Lee, Kyoung Rang (이경랑)

The Effects of a Rubric on Inexperienced Raters’ Scoring Consistency

Modern English Education
Abbr : MEESO
2016, 17(2), pp.75-90
Publisher : The Modern English Education Society
Research Area : Humanities > English Language and Literature > English Language Teaching
Published : May 31, 2016

Lee, Kyoung Rang ¹

¹세종대학교

Accredited

ABSTRACT

This study aimed to explore whether a rubric was effective for promoting raters’ scoring consistency when evaluating English learners’ essays. The experienced raters’ scores of Korean university students, with a rubric, were analyzed qualitatively and quantitatively. The scores of inexperienced raters with the rubric were compared with those without the rubric. Each group’s scores were compared using t-tests (also with inter-rater reliability tests) and the inexperienced groups’ scores were compared using ANOVA. With the rubric, the inexperienced raters showed a similar way of evaluating students’ performance to the experienced raters while those without the rubric could not. The former scored the essays consistently both within and across the semesters, but the latter could do so only within the semesters (post-test scores were higher than pre-test scores within each semester). The scores of each test given by the inexperienced raters with the rubric were not statistically different, which was not true for the experienced with the rubric. This implies that rubrics are effective in promoting inexperienced raters’ scoring consistency, especially when raters evaluate essays at the same time rather than when evaluating at different times even with a rubric. The details of the results are given, and the significance is discussed.

KEYWORDS

rubric, scoring consistency, intra-rater reliability

Citation status

* References for papers published after 2023 are currently being built.

[journal] Bacha, N. / 2001 / Writing evaluation: What can analytic versus holistic essay scoring tell us? / System 29(3) : 371~383

[journal] Beyreli, L. / 2009 / The use of analytic rubric in the assessment of writing performance: Inter-rater concordance study / Educational Sciences : Theory and Practice 9(1) : 105~125

[journal] Brown, A. / 1995 / The effect of rater variables in the development of an occupation-specific language performance test / Language Testing 12(1) : 1~15

[journal] Brown, J. D. / 1991 / Do English and ESL faculties rate writing samples differently? / TESOL Quarterly 25(4) : 587~603

[report] Cho, Seokhoon / 2008 / A study on introducing a new system of English speaking lecturers to schools / Ministry of Education

[journal] 최연희 / 2002 / FACETS Analysis of Effects of Rater Training on Secondary School English Teachers' Scoring of English Writing / 응용언어학 / 한국응용언어학회 18(1) : 13~292

[book] Hamp-Lyons, L. / 1991 / Assessing second language writing in academic contexts / Ablex Publishing : 5~15

[journal] Hyland, K. / 2006 / Teachers’ perceptions of error: The effects of first language and experience / System 34(4) : 509~519

[book] Jacobs, H. L. / 1981 / Testing ESL composition: A practical approach / Newbury House Publishers

[journal] Knoch, U. / 2007 / Re-training writing raters online: How does it compare with face-to-face training? / Assessing Writing 12(1) : 26~43

[journal] Kobayashi, T. / 1992 / Native and nonnative reactions to ESL compositions / TESOL Quarterly 26(1) : 81~112

[journal] Kohn, A. / 2006 / The trouble with rubrics / English Journal 95(4) : 12~15

[journal] Kondo-Brown, K. / 2002 / A FACETS analysis of rater bias in measuring Japanese second language writing performance / Language Testing 19(1) : 3~31

[journal] 이희경 / 2009 / Native and nonnative rater behavior in grading Korean students' English essays / Asia Pacific Education Review / 교육연구소 10(3) : 387~397

[book] McNamara, T. F. / 1996 / Measuring second language performance / Longman

[journal] O’Loughlin, K. / 1992 / Do English and ESL teachers rate essays differently? / Melbourne Papers in Language Testing 1(2) : 19~44

[journal] Rezaei, A. R. / 2010 / Reliability and validity of rubrics for assessment through writing / Assessing Writing 15(1) : 18~39

[journal] Schaefer, E. / 2008 / Rater bias patterns in EFL writing assessment / Language Testing 25(4) : 465~493

[journal] Shi, L. / 2001 / Native- and nonnative-speaking EFL teachers’ evaluation of Chinese students’ English writing / Language Testing 18(3) : 303~325

[journal] 신유선 / 2010 / A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance / 영어어문교육 / 한국영어어문교육학회 16(1) : 123~142

[journal] Silvestri, L. / 2006 / Using rubrics to increase the reliability of assessment in health classes / International Electronic Journal of Health Education 9 : 25~30

[journal] Song, B. / 1996 / Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? / Journal of Second Language Writing 5(2) : 163~182

[journal] Spandel, V. / 2006 / In defense of rubrics / English Journal 96(1) : 19~22

[book] Stevens, D. D. / 2005 / Introduction to rubrics: An assessment tool to save grading time, convey effective feedback and promote student learning / Stylus

[journal] Weigle, S. C. / 1998 / Using FACETS to model rater training effects / Language Testing 15(2) : 263~287

[book] Weigle, S. C. / 2002 / Assessing writing / Cambridge University Press

[journal] Wigglesworth, G. / 1993 / Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction / Language Testing 10(3) : 305~335

[journal] Wolf, K. / 2007 / The role of rubrics in advancing and assessing student learning / The Journal of Effective Teaching 7(1) : 3~14

KJCKorea
Journal Central

Modern English Education 2023 KCI Impact Factor : 1.55