Modern English Education 2023 KCI Impact Factor : 1.55
-
pISSN : 1598-0782 / eISSN : 2586-6141
- https://journal.kci.go.kr/meeso
KJCKorea
Journal Central
Exploring the Linguistic and Textual Factors That Affect the Quality of L2 Writing: The Coh-Metrix Study
- Modern English Education
- Abbr : MEESO
- 2023, 24(), pp.110-126
- DOI : 10.18095/meeso.2023.24.1.110
- Publisher : The Modern English Education Society
- Research Area : Humanities > English Language and Literature > English Language Teaching
- Received : May 24, 2023
- Accepted : September 10, 2023
- Published : September 10, 2023
Inam Ullah 1, Hyojung Lim 2
1Kwangwoon University
2광운대학교
Accredited
ABSTRACT
Efforts have been made to improve the output of automated essay evaluation systems to emulate human ratings in the field of applied linguistics. The current study aimed to explore the textual features that distinguish good from poor second-language (L2) writing using Coh-Metrix version 3.0 and to compare the output of an online text analysis tool to human rating. Sixty essays were collected from tertiary-level on-campus essay writing contests in 2017 - 2018. Three experienced English instructors rated the essays using a rubric from Jacobs et al. (1981). The Coh-Metrix output, including indices of descriptive, text easability, lexical diversity, connectives, latent semantic analysis (LSA), word information, and L2 readability, were evaluated by correlation analyses and independent t-tests. The results showed that higher scorers wrote longer, with a greater number of sentences and paragraphs. They were also likely to use more concrete words, more frequent words, and better cohesion, all of which contributed to readability. The pedagogical implications and limitations of using Coh-Metrix for L2 writing testing and instruction were further discussed.
KEYWORDS
Statistics
-
84 Viewed
-
55 Downloaded
Tools
Issue List
Citation status
* References for papers published after 2023 are currently being built.
[journal]
안수진
/ 2020
/
Discriminating L2 Writing Proficiency through the Use of Computational Tool Coh-Metrix
/ 언어과학연구
/ 언어과학회
(92)
: 209~232
[journal]
Aryadoust, V.
/ 2015
/ Predicting EFL writing ability from levels of mental representation measured by Coh-Metrix : A structural equation modeling study
/ Assessing Writing
24
: 35~58
[journal]
Blake, R. J.
/ 2009
/ The use of technology for second language distance learning
/ The Modern Language Journal
93
: 822~835
[book]
Blass, L.
/ 2007
/ Mosaic one: Writing paragraph review and essay development
/ McGraw-Hill
[journal]
Crossley, S. A.
/ 2016
/ The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality
/ Journal of Second Language Writing
32
: 1~16
[journal]
Crossley, S. A.
/ 2009
/ Computational assessment of lexical differences in L1 and L2 writing
/ Journal of Second Language Writing
18(2)
: 119~135
[journal]
Crossley, S. A.
/ 2011
/ Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing
/ International Journal of Continuing Engineering Education and Life-Long Learning
21(2/3)
: 170~191
[journal]
Crossley, S. A.
/ 2019
/ Moving beyond classic readability formulas: New methods and new models
/ Journal of Research in Reading
42(3-4)
: 541~561
[journal]
Crossley, S. A.
/ 2014
/ What's so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing
/ Reading in a Foreign Language
26(1)
: 92~113
[journal]
Cumming, A.
/ 2002
/ Decision making while rating ESL/EFL writing tasks : A descriptive framework
/ The Modern Language Journal
86(1)
: 67~96
[journal]
Dale, E.
/ 1948
/ A formula for predicting readability : Instructions
/ Educational Research Bulletin
27(1)
: 37~54
[journal]
Evangelopoulos, N.
/ 2012
/ Latent semantic analysis : Five methodological recommendations
/ European Journal of Information Systems
21(1)
: 70~86
[journal]
Ferrari, M.
/ 1998
/ What makes a good writer? Differences in good and poor writers' self-regulation of writing
/ Instructional Science
26
: 473~488
[journal]
Ferris, D.
/ 2001
/ Error feedback in L2 writing classes
/ Journal of Second Language Writing
10(3)
: 161~184
[journal]
Ferster, B.
/ 2012
/ Automated formative assessment as a tool to scaffold student documentary writing
/ Journal of Interactive Learning Research
23(1)
: 81~99
[journal]
Goh, T. T.
/ 2020
/ Microfeatures influencing writing quality : The case of Chinese students’ SAT essay
/ Computer Assisted Language Learning
33(4)
: 455~481
[journal]
구지혜
/ 2012
/
Transfer of L1 writing proficiency among EFL students: Testing the threshold hypothesis with Korean 9^th grade EFL learners
/ 언어학연구
/ 한국언어연구학회
17(2)
: 135~156
[journal]
Graesser, A. C.
/ 2014
/ Coh-Metrix measures text characteristics at multiple levels of language and discourse
/ The Elementary School Journal
115(2)
: 210~229
[journal]
Graesser, A. C.
/ 2004
/ Coh-Metrix: Analysis of text on cohesion and language
/ Behavior Research Methods, Instruments, & Computers
36(2)
: 193~202
[web]
Hadjerrouit, S
/ 2020
/ Exploring the affordances of Numbas for mathematical learning: A case study
/ DRUM
/ https://hal.science/hal-03113967v1/document
[book]
Halliday, M. A. K.
/ 2014
/ Cohesion in English
/ Routledge
[journal]
Hongwei, W.
/ 2013
/ A computational analysis of textual features and L2 writing proficiency
/ International Journal of Academic Research in Progressive Education and Development
2(4)
: 170~185
[journal]
Housen, A.
/ 2009
/ Complexity, accuracy, and fluency in second language acquisition
/ Applied Linguistics
30(4)
: 461~473
[journal]
Intaraprawat, P.
/ 1995
/ The use of metadiscourse in good and poor ESL essays
/ Journal of Second Language Writing
4(3)
: 253~272
[book]
Jacobs, H. L.
/ 1981
/ Testing ESL composition: A practical approach
/ Newbury House
[journal]
정연주
/ 2019
/
Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools
/ Journal of Asia TEFL
/ 아시아테플
16(1)
: 37~52
[journal]
Kobrin, J. L.
/ 2011
/ Contextual factors associated with the validity of SAT scores and high school GPA for predicting first-year college grades
/ Educational Assessment
16(4)
: 207~226
[journal]
공은정
/ 2013
/
L2 Proficiency Effect on the Acoustic Cue-Weighting Pattern by Korean L2 Learners of English:Production and Perception of English Stops
/ 말소리와 음성과학
/ 한국음성학회
5(4)
: 81~90
[journal]
Larsen-Freeman, D.
/ 2006
/ The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English
/ Applied Linguistics
27(4)
: 590~619
[journal]
이은희
/ 2014
/
An Analysis of Korean EFL University Learners’ Use of Cohesion in Writing with Coh-Metrix
/ 외국학연구
/ 외국학연구소
(27)
: 195~216
[journal]
Lee, J-W.
/ 1997
/ The relative contribution of L2 language proficiency and L1 reading ability to L2 reading performance : A test of the threshold hypothesis in an EFL context
/ TESOL Quarterly
31(4)
: 713~739
[journal]
Lentz, L.
/ 1997
/ The evaluation of text quality : Expert-focused and reader-focused methods compared
/ IEEE Transactions on Professional Communication
40(3)
: 224~234
[journal]
임효정
/ 2023
/
Exploring the Development of Second Language (L2) Vocabulary in Speaking
/ 외국어교육연구
/ 외국어교육연구소
37(1)
: 145~163
[journal]
Lim, H.
/ 2014
/ Automatization in second language sentence processing : A partial, conceptual replication of Hulstijn, Van Gelderen, and Schoonen’s 2009 study
/ Applied Psycholinguistics
36(5)
: 1247~1282
[journal]
Lim, H.
/ 2012
/ Review of criterion for English language learning
/ Language Learning & Technology
16(2)
: 38~45
[journal]
Lu, X.
/ 2010
/ Automatic analysis of syntactic complexity in second language writing
/ International Journal of Corpus Linguistics
15(4)
: 474~496
[journal]
MacArthur, C. A.
/ 2018
/ Which linguistic features predict quality of argumentative writing for college basic writers, and how do those features change with instruction?
/ Reading and Writing
32(6)
: 1553~1574
[book]
Machicao, J. C
/ 2019
/ Information technology and systems:Advances in intelligent systems and computing
/ Springer
: 835~844
[thesis]
Malmcrona, H
/ 2020
/ Variation in assessment: A Coh-Metrix analysis of evaluation of written English in the Swedish upper secondary school
/ master
/ Stockholm University
[journal]
McCarthy, P. M.
/ 2010
/ MTLD, vocd-D, and HD-D : A validation study of sophisticated approaches to lexical diversity assessment
/ Behavior Research Methods
42(2)
: 381~392
[book]
McNamara, D. S.
/ 2012
/ Applied natural language processing and content analysis: Identification, investigation and resolution
/ IGI Global
: 188~205
[journal]
McNamara, D. S.
/ 2010
/ Coh-Metrix : Capturing linguistic features of Cohesion
/ Discourse Processes
47(4)
: 292~330
[book]
Mody, M. E.
/ 2008
/ Brain, behavior, and learning in language and reading disorders
/ Guilford Press
[journal]
박묘영
/ 2012
/
Exploring the Raters’ Bias on an EFL Writing Assessment Using Multi-faceted Rasch Measurement
/ Studies in English Education
/ 글로벌영어교육학회
17(2)
: 175~202
[journal]
Petchprasert, A.
/ 2021
/ Utilizing an automated tool analysis to evaluate EFL students’ writing performances
/ Asian-Pacific Journal of Second and Foreign Language Education
6(1)
: 1~16
[book]
Peters, E
/ 2019
/ The Routledge handbook of vocabulary studies
/ Routledge
: 125~142
[confproc]
Pitler, E.
/ 2008
/ Revisiting readability: A unified framework for predicting text quality
/ Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing
/ Association for Computational Linguistics
: 86~195
[book]
Plag, I.
/ 2015
/ Introduction to English Linguistics
/ Walter de Gruyter GmbH & Co KG
[journal]
Plakans, L.
/ 2017
/ An assessment perspective on argumentation in writing
/ Journal of Second Language Writing
36
: 85~86
[journal]
Rao, P. S.
/ 2019
/ The significance of writing skills in ELL environment
/ Academician : An International Multidisciplinary Research Journal
9(3)
: 5~17
[journal]
Sawaki, Y.
/ 2013
/ Understanding learner strengths and weaknesses : Assessing performance on an integrated writing task
/ Language Assessment Quarterly
10(1)
: 73~95
[journal]
Schaefer, E.
/ 2008
/ Rater bias patterns in an EFL writing assessment
/ Language Testing
25(4)
: 465~493
[journal]
Schmitt, N.
/ 2008
/ Instructed second language vocabulary learning
/ Language Teaching Research
12(3)
: 329~363
[journal]
Shermis, M. D.
/ 2014
/ State-of-the-art automated essay scoring : Competition, results, and future directions from a United States demonstration
/ Assessing Writing
20
: 53~76
[book]
Shiroza, S.
/ 2020
/ Functional Variations in English: Theoretical Considerations and Practical Challenges
/ Springer
: 277~293
[journal]
Snow, E. L.
/ 2015
/ Keys to detecting writing flexibility over time : entropy and natural language processing
/ Journal of Learning Analytics
2(3)
: 40~54
[journal]
Sukandi, S. S.
/ 2018
/ EFL students’ responses to learning basic reading and writing skills
/ Studies in English Language and Education
5(1)
: 40~53
[book]
Vygotsky, L. S.
/ 1978
/ Mind in society: Development of higher psychological processes
/ Harvard university press
[journal]
Wang, Q.
/ 2009
/ Investigating critical thinking and knowledge construction in an interactive learning environment
/ Interactive Learning Environments
17(1)
: 95~104
[journal]
Weigle, S. C.
/ 2007
/ Teaching writing teachers about assessment
/ Journal of Second Language Writing
16(3)
: 194~209
[journal]
Winke, P.
/ 2015
/ ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study
/ Assessing Writing
25
: 38~54
[journal]
Zaytseva, V.
/ 2019
/ The effects of contexts on the acquisition of oral lexical ability in English as a foreign language
/ The Language Learning Journal
49(5)
: 597~613