Exploring the Role of Automatically-derived Text Complexity Features in L2 Reading Test Development

Sumi Han (한수미); Jeong-Ah Shin (신정아)

Exploring the Role of Automatically-derived Text Complexity Features in L2 Reading Test Development

Modern English Education
Abbr : MEESO
2016, 17(2), pp.1-19
Publisher : The Modern English Education Society
Research Area : Humanities > English Language and Literature > English Language Teaching
Published : May 31, 2016

Sumi Han ¹, Jeong-Ah Shin ²

¹서울대학교
²동국대학교

Accredited

ABSTRACT

Although automatic text analysis tools are available, little research has been conducted on the application of such tools in reading assessments. When the ratio of academic vocabulary and transitions are computed automatically and used in test development, the text selection-revision procedure can be fast and transparent by complementing test developers’ expertise. To obtain empirical evidence for the utility of automatic text complexity features, this study attempted to explore the role of automatically-derived text complexity features in an intensive English program (IEP) reading assessment. Based on previous literature and the testing context, a total of 11 text complexity features as lexical, syntactic, and semantic variables were chosen, and their accountability for the IEP reading item difficulty was automatically measured by using three text analysis tools—Lexile, the Compleat Lexical Tutor, and Coh-Metrix. Results showed that seven complexity features significantly correlated with the reading item difficulty. Stepwise multiple regressions showed that a set of four lexical and semantic text complexity features (i.e., word length, total word counts, Lexical Semantic Analysis (LSA), connectives) explained about 45% of the variance in the reading item difficulty. The results and findings of this study are discussed with regard to limitations and implications for both reading assessments and instruction.

KEYWORDS

reading test development, text complexity, automated text analysis

Citation status

* References for papers published after 2023 are currently being built.

[book] Alderson, C. C. / 2000 / Assessing reading / Cambridge University Press

[book] Castello, E. / 2008 / Text complexity and reading comprehension tests / Peter Lang

[web] Cobb, T. / The Compleat Lexical Tutor / http://www.lextutor.ca

[journal] Crossley, S. A. / 2008 / Assessing text readability using cognitively based indices / TESOL Quarterly 43(3) : 475~492

[journal] Drum, P. A. / 1981 / The effects of surface structure variables on performance in reading comprehension tests / Reading Research Quarterly 16(4) : 486~514

[journal] Embretson, S. E. / 1987 / Component latent trait models for paragraph comprehension tests / Applied Psychological Measurement 11(2) : 175~193

[journal] Fisher, D. / 2014 / Addressing CCSSO anchor standard 10: Text complexity / Language Arts 91(4) : 250~250

[book] Fisher, D. / 2012 / Text complexity: Raising rigor in reading / International Reading Association

[journal] Freedle, R. / 1993 / The prediction of TOEFL reading item difficulty: Implications for construct validity / Language Testing 10(2) : 133~170

[journal] Graesser, A. C. / 2004 / Coh-Metrix: Analysis of text on cohesion and language / Behavior Research Methods, Instruments, & Computers 36(2) : 193~202

[book] Gray, W. S. / 1935 / What makes a book readable? / University of Chicago Press

[report] IBM Corp / 2013 / IBM SPSS Statistics for Windows, Version 22.0 / IBM Corp

[journal] 전문기 / 2015 / An analysis of the continuity among middle school English textbooks with an automated language analysis program. / 현대영어교육 / 현대영어교육학회 16(1) : 195~218

[journal] 전문기 / 2009 / A Corpus-based analysis of middle school English 1 textbooks with Coh-Metrix / 영어교육연구 / 팬코리아영어교육학회 21(4) : 265~292

[journal] 김재은 / 2015 / A corpus-based comparative analysis of linguistic difficulty among high school English textbooks, EBS-CSAT prep books, and College Scholastic Ability Test. / 멀티미디어 언어교육 / 한국멀티미디어언어교육학회 18(1) : 59~92

[book] Livingston, S. A. / 2013 / Delta equating / Educational Testing Service

[book] McNamara, D. S. / 2012 / Assessing reading in the 21th century: Aligning and applying advances in the reading and measurement sciences / R & L Education : 89~116

[journal] McNamara, D. S. / 1996 / Learning from text: Effects of prior knowledge and text coherence / Discourse Processes 22(3) : 247~287

[web] National Governors Center for Best Practices / 2010 / Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects: Appendix A: Research supporting key elements of the standards and glossary of key terms / http://www.corestandards.org/assets/Appendix_A.pdf

[book] Tabachnick, B. G. / 2007 / Using multivariate statistics / Pearson Education, Inc

KJCKorea
Journal Central

Modern English Education 2023 KCI Impact Factor : 1.55