Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

Sookyung Cho (조수경); Park, Chanho (박찬호)

doi:10.18095/meeso.2021.22.4.12

Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

Modern English Education
Abbr : MEESO
2021, 22(4), pp.12-23
DOI : 10.18095/meeso.2021.22.4.12
Publisher : The Modern English Education Society
Research Area : Humanities > English Language and Literature > English Language Teaching
Received : September 30, 2021
Accepted : November 5, 2021
Published : November 30, 2021

Sookyung Cho ¹, Park, Chanho ²

¹한국외국어대학교
²계명대학교

Accredited

ABSTRACT

The aim of this study was to explore the possibility of diagnosing second language (L2) learners’ writing skills through a multiple-choice examination by applying a CDM (cognitive diagnosis model). L2 learners’ writing abilities have mostly been assessed using direct methods such as essay tests. However, this method not only requires a large amount of human raters’ labor, but also faces several issues such as intra-rater and inter-rater reliabilities. Essay examinations are also hard to validate because they are limited in checking the internal structure (i.e., conducting factor analysis). To compensate for these limitations, this study developed a multiple-choice examination and validated it with factor analysis and a CDM. The examination consisted of 20 items on key L2 writing components: content, organization, grammar, vocabulary, and mechanics. It was conducted on 109 college students. The CDM analysis revealed that the test was valid based on response processes. It provided substantial information about each test-taker’s various skills. These results imply that L2 learners’ writing abilities can be assessed in an indirect way and that the test can be conducted with a large body of stu-dents to provide useful information regarding their writing abilities at the same time.

KEYWORDS

multiple-choice exam, CDM, second language writing, validity, reliability

Citation status

* References for papers published after 2023 are currently being built.

[journal] Ackerman, T. A. / 1988 / A comparison of the information provided by essay, multiple-choice, and free-response writing test / Applied Psychological Measurement 12(2) : 117~128

[book] Akaike, H. / 1973 / Second international symposium on information theory / Akademai Kiado : 267~281

[other] American Educational Research Association / 2014 / Standards for educational and psychological testing / American Educational Research Association

[journal] Behizadeh, N. / 2011 / Historical view of the influences of measurement and writing theories on the practice of writing assessment in the United States / Assessing Writing 16(3) : 189~211

[journal] Bejar, I. / 2012 / Rater cognition : Implications for validity / Educational Measurement : Issues and Practice 16(3) : 189~211

[journal] Bozdogan, H. / 1987 / Model selection and Akaike’s Information Criterion (AIC): The general theory and its analytical extensions / Psychometrika 52(3) : 345~370

[journal] de la Torre, J. / 2009 / DINA model and parameter estimation : A didactic / Journal of Educational and Behavioral Statistics 34(1) : 115~130

[journal] de la Torre, J. / 2011 / The generalized DINA model framework / Psychometrika 76(2) : 179~199

[book] Eckes, T. / 2016 / Handbook of second language assessment / De Gruyter Mouton : 147~164

[book] Giraud, C. / 2015 / Introduction to high-dimensional statis tics / CRC

[thesis] Hartz, S. M. / 2002 / A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality / doctoral / University of Illinois at Urbana-Champaign

[journal] Huddleston, E. M. / 1954 / Measurement of writing ability at the college-entrance level : Objective vs. subjective testing techniques / Journal of Experimental Education 22(3) : 165~213

[journal] Kim, Youn-Hee / 2011 / Diagnosing EAP writing ability using the reduced reparameterized unified model / Language Testing 28(4) : 509~541

[journal] Knoch, U. / 2011 / Investigating the effectiveness of individualized feedback to rating behaviour : A longitudinal study / Language Testing 28(2) : 179~200

[book] Lumley, T. / 2005 / Assessing second language writing: The rater’s perspective / Lang

[journal] Lumley, T. / 1995 / Rater characteristics and rater bias: Implications for training / Language Testing 12(1) : 54~71

[journal] Ma, W. / 2020 / GDINA : An R package for cognitive diagnosis modeling / Journal of Statistical Software 93(14) : 1~26

[journal] Shaun J. Manning / 2019 / Engagement With Automated Writing Feedback in Mandated vs. Voluntary Conditions / 현대영어교육 / 현대영어교육학회 20(4) : 18~30

[book] McDonald, R. P. / 1999 / Test theory: A unified treatment / Lawrence Erlbaum Associates Publishers

[book] Muthén, L. K. / 2017 / Mplus user’s guide / Muthén & Muthén

[journal] Palmer, E. J. / 2007 / Assessment of higher order cognitive skills in undergraduate education:Modified essay or multiple choice questions? Research paper / BMC Medical Education 7(49) : 1~7

[journal] 박찬호 / 2011 / Cognitive Diagnostic Assessment of English Grammar for Korean EFL Learners / 영어교육 / 한국영어교육학회 66(4) : 101~117

[journal] 박찬호 / 2017 / An Exploratory Analysis of Compensatory Cognitive Diagnosis in EFL Reading Comprehension / 영어학 / 한국영어학회 17(1) : 85~104

[journal] 박찬호 / 2020 / Cognitive Diagnostic Writing Assessment for Korean Learners of English / 외국어교육연구 / 외국어교육연구소 34(2) : 1~24

[journal] Justin Paulsen / 2021 / Examining cognitive diagnostic modeling in classroom assessment conditions / The Journal of Experimental Education / Informa UK Limited : 1~18

[web] R Core Team / 2021 / R: A language and environment for statistical computing / R Foundation for Statistical Computing / https://www.R-project.org/

[web] Robitzsch, A. / 2020 / CDM: Cognitive diagnosis modeling. R package version 7.5-15 / https://CRAN.R-project.org/package=CDM

[book] Rupp, A. A. / 2010 / Diagnostic measurement: Theory, methods, and applications / Guilford

[journal] Sawaki, Y. / 2009 / Q-matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessment / Language Assessment Quarterly 6(3) : 190~209

[journal] Schwarz, G. / 1978 / Estimating the dimension of a model / Annals of Statistics 6(2) : 461~464

[book] von Davier, M. / 2019 / Handbook of diagnostic classification models: Models and model extensions, applications, software packages / Springer

[book] Wall, D. / 2012 / The Routledge handbook of language testing / Routledge : 79~92

[journal] Weigle, S. / 1998 / Using FACETS to model rater training effects / Language Testing 15(2) : 263~287

[journal] Weigle, S. / 1999 / Investigating rater/prompt interactions in writing assessment : Quantitative and qualitative approaches / Assessing Writing 6(2) : 145~178

[journal] Xie, Q. / 2016 / Diagnosing university students’ academic writing in English: Is cognitive diagnostic modelling the way forward? / Educational Psychology 37(1) : 26~47

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Modern English Education 2023 KCI Impact Factor : 1.55

Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Modern English Education 2023 KCI Impact Factor : 1.55

Development of a Multiple-Choice Writing Examination Validated with a Cognitive Diagnosis Model

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (2)

REFERENCES (35) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.