A Comparative Analysis of Inter-rater Reliability in English Essay and Korean-English Translation Tests

Park， Lili (박리리); Choi, Inn-Chull (최인철)

A Comparative Analysis of Inter-rater Reliability in English Essay and Korean-English Translation Tests

Modern English Education
Abbr : MEESO
2016, 17(1), pp.27-48
Publisher : The Modern English Education Society
Research Area : Humanities > English Language and Literature > English Language Teaching
Published : February 28, 2016

Park， Lili ¹, Choi, Inn-Chull ²

¹한세대학교
²고려대학교

Accredited

ABSTRACT

Writing performance assessment is gaining momentum in the field of English education in Korea. Writing performance tests, however, are prone to subjective rating that may have adverse effects on the reliability of scores. Previous research recommended reinforced training and clearer scoring criteria. Rater training, however, is costly and demanding in the case of essay tests which require understanding and application of multiple criteria without much guidance on benchmark responses. This study thus sought to examine translation as an alternative writing task that may induce greater score reliability as well as accurate measurement of writing proficiency. Three raters scored the responses of 20 Korean college students to an English to Korean translation test and an English essay test solely based on the rubrics. The results showed that inter-rater reliability was higher for the translation test than for the essay test. In addition, the agreement between two of the raters reached a statistically significant level for the translation test. The implications of these findings are discussed in relation to the viability of implementing translation tests in secondary schools together with suggestions for further study on ways to strengthen the advantages of both essay and translation tests and make up for their weaknesses.

KEYWORDS

writing proficiency, English writing test, English essay, Korean-English translation, inter-rater reliability

Citation status

* References for papers published after 2023 are currently being built.

[journal] Ahmed, A. / 2011 / Improving marking quality through a taxonomy of mark schemes / Assessment in Education: Principles, Policy & Practice 18(3) : 259~278

[journal] 안경자 / 2009 / Learning to Write Academic English: A Sociocultural Perspective on L2 Learners’ Writing Portfolios / 응용언어학 / 한국응용언어학회 25(2) : 1~34

[book] Alderson, J. / 2002 / Language test construction and evaluation / Cambridge University Press

[book] Bachman, L. F. / 1990 / Fundamental considerations in language testing / Oxford University Press

[book] Bachman, L. F. / 1996 / Language testing in practice / Oxford University Press

[journal] Brown, A. / 2003 / Interviewer variation and co-construction of speaking proficiency / Language Testing 20 : 1~25

[book] Brown, G. T. L. / 2009 / Tertiary assessment and higher education student outcomes: Policy, practice and research / Ako Aotearoa : 40~48

[journal] 조동완 / 2009 / 번역 과제 발표 중심의 대학 영작문 수업에 관한 연구 / Foreign Languages Education / 한국외국어교육학회 16(1) : 261~289

[book] Choi, Inn-Chull / 2010 / Silyong Yeongmunbeob Baeggwasajeon [Practical English grammar encyclopedia] / SaramIn Publishing Co

[journal] Choi, Inn-Chull / 2001 / Comparative analyses of computer-based ratings and human ratings based on performance testing of writing / Multimedia-Assisted Language Learning 4(1) : 165~184

[journal] 전유아 / 2009 / Collocations in L2 Writing and Rater's Perceived Writing Proficiency / 응용언어학 / 한국응용언어학회 25(1) : 101~129

[report] Chung, G. / 1997 / Methodological approaches to online scoring of essays / National Center for Research on Evaluation, Standards, and Student Testing

[journal] Cohen, A. D. / 2001 / Research on direct versus translated writing:Students’ strategies and their results / The Modern Language Journal 85(2) : 169~188

[journal] Cumming, A. / 1989 / Writing expertise and second language proficiency / Language Learning 39 : 81~141

[journal] Eckes, T. / 2008 / Rater types in writing performance assessments: A classification approach to rater variability / Language Testing 25(2) : 155~185

[journal] Eckes, T. / 2012 / Operational rater types in writing assessment: Linking rater cognition to rater behavior / Language Assessment Quarterly 9(3) : 317~334

[journal] Enright, M. K. / 2010 / Complementing human judgment of essays written by English learners with e-rater ® scoring / Language Testing 27(3) : 317~334

[journal] Fritz E. / 2013 / Rater sensitivity to lexical accuracy, sophistication and range when assessing writing / Assessing Writing 18(2) : 173~181

[journal] Garcia, I. / 2011 / Machine translation-assisted language learning: Writing for beginners / Computer Assisted Language Learning: An International Journal 24(5) : 471~487

[journal] Guo, L. / 2013 / Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study / Assessing Writing 18(3) : 218~238

[journal] He, T. H. / 2013 / Multi-faceted Rasch measurement and bias patterns in EFL writing performance assessment / Psychological Reports 112(2) : 469~485

[journal] Kyung-AeJin / 2002 / Applying Standards-Based English Performance Assessment in Korean High Schools / 현대문법연구 / 현대문법학회 (28) : 231~244

[journal] Kane, M. T. / 1999 / Validating measures of performance / Educational Measurement: Issues and Practice 18(2) : 5~17

[journal] 김인숙 / 2011 / Paper-based versus Image-based Scoring of Student Essays / 응용언어학 / 한국응용언어학회 27(2) : 231~256

[journal] 김정아 / 2010 / Grammatical and Discursive Features of Korean EFL Learners’ Direct, Translated and Back-translated Writing / 언어연구 / 언어정보연구소 27(2) : 373~392

[journal] 김진아 / 2002 / A Study on Quality Assessment Criteria in English-Korean Translation / 외국문학연구 / 외국문학연구소 (11) : 4~124

[journal] 김연희 / 2013 / The effect of different types of rating scales on ESL writing performance assessment. / 영어학 / 한국영어학회 13(3) : 465~496

[journal] Kim, Young-Ran / 2014 / The use of L1 as a writing strategy in L2 writing tasks / GEMA Online Journal of Language Studies 14(3) : 33~50

[journal] 이상기 / 2013 / Different Rater Groups and Their Feedback on Written Texts: Where Do They Focus Their Attention? / 영어교과교육 / 한국영어교과교육학회 12(1) : 57~76

[journal] 이요안 / 2009 / Reevaluating Advanced Speaking Proficiency for Korean Teachers of English: A Case Study / 응용언어학 / 한국응용언어학회 25(3) : 303~336

[journal] Lind Pantzare, A. / 2015 / Interrater reliability in large-scale assessments: Can teachers score national tests reliably without external controls? / Practical Assessment, Research & Evaluation 20(9)

[journal] Lumley, T. / 2002 / Assessment criteria in a large-scale writing test: What do they really mean to the raters? / Language Testing 19(3) : 246~276

[journal] Lumley, T. / 1995 / Rater characteristics and rater bias: Implications for training / Language Testing 12 : 54~71

[book] McNamara, T. / 1996 / Measuring second language performance / Longman

[book] Nida, E. A. / 1982 / The theory and practice of translation / Koninklijke Brill NV

[journal] 박지선 / 2012 / A report on the teaching and learning of speaking and writing in high school English classrooms: Transiting to NEAT / 현대영어교육 / 현대영어교육학회 13(2) : 121~149

[journal] Park, Young-Soon / 2002 / The basic theory of interpretation and translation studies / Modern English Education 3(2) : 99~134

[journal] Plakans, L. / 2010 / Independent vs. integrated writing tasks: A comparison of task representation / TESOL Quarterly 44(1) : 185~194

[journal] Schaefer, E. / 2008 / Rater bias patterns in an EFL writing assessment / Language Testing 25(4) : 465~493

[journal] Shin, Bong-Soo / 1998 / Experimental study on the effects of Korean on English writing:Direct writing versus translation writing / Studies in Language 14 : 127~142

[journal] 신유선 / 2010 / A FACETS Analysis of Rater Characteristics and Rater Bias in Measuring L2 Writing Performance / 영어어문교육 / 한국영어어문교육학회 16(1) : 123~142

[journal] 신유선 / 2010 / Investigating Variability in Tasks, Analytic Scoring Criteria, and Rater Groups on L2 Writing in English / Foreign Languages Education / 한국외국어교육학회 17(2) : 27~57

[journal] Tavakoli, M. / 2014 / Direct versus translated writing: The effect of translation on learners’ second language writing ability / GEMA Online Journal of Language Studies 14(2) : 61~74

[journal] Uzawa, K. / 1989 / Writing strategies in Japanese as a foreign language:Lowering or keeping up the standards / The Canadian Modern Language Review 46 : 178~194

[journal] Veerappan, V. / 2012 / A review on IELTS writing test, its test results and inter rater reliability / Theory & Practice in Language Studies 2(1) : 138~143

[journal] Weigle, S. / 1994 / Effects of training on raters of ESL compositions / Language Testing 11(1) : 197~223

[journal] Weigle, S. / 1998 / Using FACETS to model rater training effects / Language Testing 15 : 263~287

[journal] Weigle, S. C. / 2003 / Effects of task and rater background on the evaluation of ESL student writing: A pilot study / TESOL Quarterly 37(2) : 345~354

KJCKorea
Journal Central

Modern English Education 2023 KCI Impact Factor : 1.55