Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

Sookyung Cho (조수경); Park, Chanho (박찬호)

doi:10.18095/meeso.2021.22.4.12

Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

Modern English Education
Abbr : MEESO
2021, 22(4), pp.12~23
DOI : 10.18095/meeso.2021.22.4.12
Publisher : The Modern English Education Society
Research Area : Humanities > English Language and Literature > English Language Teaching
Received : September 30, 2021
Accepted : November 5, 2021
Published : November 30, 2021

Sookyung Cho ¹, Park, Chanho ²

¹한국외국어대학교
²계명대학교

Accredited

ABSTRACT

The aim of this study was to explore the possibility of diagnosing second language (L2) learners’ writing skills through a multiple-choice examination by applying a CDM (cognitive diagnosis model). L2 learners’ writing abilities have mostly been assessed using direct methods such as essay tests. However, this method not only requires a large amount of human raters’ labor, but also faces several issues such as intra-rater and inter-rater reliabilities. Essay examinations are also hard to validate because they are limited in checking the internal structure (i.e., conducting factor analysis). To compensate for these limitations, this study developed a multiple-choice examination and validated it with factor analysis and a CDM. The examination consisted of 20 items on key L2 writing components: content, organization, grammar, vocabulary, and mechanics. It was conducted on 109 college students. The CDM analysis revealed that the test was valid based on response processes. It provided substantial information about each test-taker’s various skills. These results imply that L2 learners’ writing abilities can be assessed in an indirect way and that the test can be conducted with a large body of stu-dents to provide useful information regarding their writing abilities at the same time.

KEYWORDS

multiple-choice exam, CDM, second language writing, validity, reliability

Journal Copyright Policy

This license enables reusers to distribute, remix, adapt, and build upon the material in any medium or format for noncommercial purposes only, and only so long as attribution is given to the creator.

Citation status

* References for papers published after 2025 are currently being built.

[journal] Ackerman, T. A. / 1988 / A comparison of the information provided by essay, multiple-choice, and free-response writing test / Applied Psychological Measurement 12(2) : 117~128

[book] Akaike, H. / 1973 / Second international symposium on information theory / Akademai Kiado : 267~281

[other] American Educational Research Association / 2014 / Standards for educational and psychological testing / American Educational Research Association

[journal] Behizadeh, N. / 2011 / Historical view of the influences of measurement and writing theories on the practice of writing assessment in the United States / Assessing Writing 16(3) : 189~211

[journal] Bejar, I. / 2012 / Rater cognition : Implications for validity / Educational Measurement : Issues and Practice 16(3) : 189~211

[journal] Bozdogan, H. / 1987 / Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions / Psychometrika 52(3) : 345~370

[journal] de la Torre, J. / 2009 / DINA model and parameter estimation : A didactic / Journal of Educational and Behavioral Statistics 34(1) : 115~130

[journal] de la Torre, J. / 2011 / The generalized DINA model framework / Psychometrika 76(2) : 179~199

[book] Eckes, T. / 2016 / Handbook of second language assessment / De Gruyter Mouton : 147~164

[book] Giraud, C. / 2015 / Introduction to high-dimensional statis tics / CRC

[thesis] Hartz, S. M. / 2002 / A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality / doctoral / University of Illinois at Urbana-Champaign

[journal] Huddleston, E. M. / 1954 / Measurement of writing ability at the college-entrance level : Objective vs. subjective testing techniques / Journal of Experimental Education 22(3) : 165~213

[journal] Kim, Youn-Hee / 2011 / Diagnosing EAP writing ability using the reduced reparameterized unified model / Language Testing 28(4) : 509~541

[journal] Knoch, U. / 2011 / Investigating the effectiveness of individualized feedback to rating behaviour : A longitudinal study / Language Testing 28(2) : 179~200

[book] Lumley, T. / 2005 / Assessing second language writing: The rater’s perspective / Lang

[journal] Lumley, T. / 1995 / Rater characteristics and rater bias: Implications for training / Language Testing 12(1) : 54~71

[journal] Ma, W. / 2020 / GDINA : An R package for cognitive diagnosis modeling / Journal of Statistical Software 93(14) : 1~26

[journal] Shaun J. Manning / 2019 / Engagement With Automated Writing Feedback in Mandated vs. Voluntary Conditions / 현대영어교육 / 현대영어교육학회 20(4) : 18~30

[book] McDonald, R. P. / 1999 / Test theory: A unified treatment / Lawrence Erlbaum Associates Publishers

[book] Muthén, L. K. / 2017 / Mplus user’s guide / Muthén & Muthén

[journal] Palmer, E. J. / 2007 / Assessment of higher order cognitive skills in undergraduate education:Modified essay or multiple choice questions? Research paper / BMC Medical Education 7(49) : 1~7

[journal] 박찬호 / 2011 / Cognitive Diagnostic Assessment of English Grammar for Korean EFL Learners / 영어교육 / 한국영어교육학회 66(4) : 101~117

[journal] 박찬호 / 2017 / An Exploratory Analysis of Compensatory Cognitive Diagnosis in EFL Reading Comprehension / 영어학 / 한국영어학회 17(1) : 85~104

[journal] 박찬호 / 2020 / Cognitive Diagnostic Writing Assessment for Korean Learners of English / 외국어교육연구 / 외국어교육연구소 34(2) : 1~24

[journal] Justin Paulsen / 2021 / Examining cognitive diagnostic modeling in classroom assessment conditions / The Journal of Experimental Education / Informa UK Limited : 1~18

[web] R Core Team / 2021 / R: A language and environment for statistical computing / R Foundation for Statistical Computing / https://www.R-project.org/

[web] Robitzsch, A. / 2020 / CDM: Cognitive diagnosis modeling. R package version 7.5-15 / https://CRAN.R-project.org/package=CDM

[book] Rupp, A. A. / 2010 / Diagnostic measurement: Theory, methods, and applications / Guilford

[journal] Sawaki, Y. / 2009 / Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessment / Language Assessment Quarterly 6(3) : 190~209

[journal] Schwarz, G. / 1978 / Estimating the dimension of a model / Annals of Statistics 6(2) : 461~464

[book] von Davier, M. / 2019 / Handbook of diagnostic classification models: Models and model extensions, applications, software packages / Springer

[book] Wall, D. / 2012 / The Routledge handbook of language testing / Routledge : 79~92

[journal] Weigle, S. / 1998 / Using FACETS to model rater training effects / Language Testing 15(2) : 263~287

[journal] Weigle, S. / 1999 / Investigating rater/prompt interactions in writing assessment : Quantitative and qualitative approaches / Assessing Writing 6(2) : 145~178

[journal] Xie, Q. / 2016 / Diagnosing university students’ academic writing in English: Is cognitive diagnostic modelling the way forward? / Educational Psychology 37(1) : 26~47

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Modern English Education 2025 KCI Impact Factor : 0.96