How the L2 LSTM language model predicts island effects

SunJoo Choi (최선주); Euhee Kim (김유희)

doi:10.33639/ptc.2021..36.019

How the L2 LSTM language model predicts island effects

PHILOSOPHY·THOUGHT·CULTURE
2021, (36), pp.411~430
DOI : 10.33639/ptc.2021..36.019
Publisher : Research Institute for East-West Thought
Research Area : Humanities > Other Humanities
Received : May 9, 2021
Accepted : June 30, 2021
Published : June 30, 2021

SunJoo Choi ¹, Euhee Kim ²

¹동국대학교
²신한대학교

Accredited

ABSTRACT

In this paper, we investigate how well L2 language models learn about particular types of controlled contexts that indicate representation of syntactic dependencies (e.g., the filler-gap dependency and possessive pronoun gender expectation). Building the three language models, we assess LM’s knowledge by comparing the interaction in island configurations to that in non-island minimal paris. In so doing, we provide new evidence that L2 models cannot significantly maintain their expectations through island constructions while L1 model is able to thread expectations within islands. Furthermore, L2 models fail to correctly learn about empty syntactic positions while L1 model is able to generalize about them. Additionally, we control the amount of training data for L2 models to examine the correlation of the LM’s performances. However, we cannot investigate the correlation between the size of sentences that L2ers learn and their sentence processing patterns. Previous study is inconclusive about whether L2ers learn these linguistic constraints. Here, using new methodology, we show evidence about the characteristics of L2ers’ behavior.

KEYWORDS

LSTM Language model, Filler-gap dependency, Possessive pronoun gender expectation, Surprisal, Syntactic islands.

Citation status

* References for papers published after 2023 are currently being built.

[journal] Baayen, R. Harald. / 2018 / Mixed-effects modeling with crossed random effects for subjects and items / Journal of memory and language 59(4) : 390~412

[journal] Bernardy, Jean-Philippe. / 2017 / Using deep neural networks to learn syntactic agreement / Linguistic Issues in Language Technology

[other] Gulordava, Kristina. / 2018 / Colorless green recurrent networks dream hierarchically / arXiv: 1803.11138

[confproc] Hale, John. / 2001 / A probabilistic Earley parser as a psycholinguistic model / Second meeting of the north american chapter of the association for computational linguistics

[journal] Hochreiter, Sepp. / 1997 / Long short-term memory / Neural computation 9(8) : 1735~1780

[other] Jozefowicz, Rafal. / 2016 / Exploring the limits of language modeling / arXiv: 1602.02410

[journal] Levy, Roger / 2008 / Expectation-based syntactic comprehension / Cognition 106(3) : 1126~1177

[journal] Linzen, Tal. / 2016 / Assessing the ability of LSTMs to learn syntax-sensitive dependencies / Transactions of the Association for Computational Linguistics 4 : 521~535

[journal] Smith, Nathaniel J. / 2013 / The effect of word predictability on reading time is logarithmic / Cognition 128(3) : 302~319

[other] Wilcox, Ethan. / 2019 / What syntactic structures block dependencies in RNN language models? / arXiv :1905.10431

[other] Wilcox, Ethan. / 2018 / What do RNN Language Models Learn about Filler-Gap Dependencies? / arXiv:1809.00042

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

PHILOSOPHY·THOUGHT·CULTURE 2023 KCI Impact Factor : 0.5

How the L2 LSTM language model predicts island effects

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

PHILOSOPHY·THOUGHT·CULTURE 2023 KCI Impact Factor : 0.5

How the L2 LSTM language model predicts island effects

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (11) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.