Application of Automatic Evaluation to Human Translation (번역자동평가에서 풀리지 않은 과제)

Hyeyeon Chung (정혜연); Kim， Bo-young (김보영); Kim， Yeon-joo (김연주); Seo， Seung-hee (서승희); Song， Shin-ae (송신애); Lee， Jin-Hyun (이진현); Jeon Kyoung-ah (전경아); CHOI JISOO (최지수); Hong， Seung-bin (홍승빈); Heo TakSung (허탁성)

doi:10.15749/jts.2020.21.1.001

Application of Automatic Evaluation to Human Translation

The Journal of Translation Studies
Abbr : JTS
2020, 21(1), pp.9-29
DOI : 10.15749/jts.2020.21.1.001
Publisher : The Korean Association for Translation Studies
Research Area : Humanities > Interpretation and Translation Studies
Received : February 10, 2020
Accepted : March 11, 2020
Published : March 31, 2020

Hyeyeon Chung ¹, Kim， Bo-young ¹, Kim， Yeon-joo ¹, Seo， Seung-hee ¹, Song， Shin-ae ¹, Lee， Jin-Hyun ¹, Jeon Kyoung-ah ¹, CHOI JISOO ¹, Hong， Seung-bin ¹, Heo TakSung ²

¹한국외국어대학교
²한림대학교

Accredited

ABSTRACT

This paper deals with two questions. The first is whether BLEU and METEOR, which were developed to evaluate machine translation, can also be used for the evaluation of human translations. The second is, how can these systems be adapted to evaluate human translations in a more variable way? These questions can be subdivided into the following questions: (1) What is the more variable evaluation system, BLEU or METEOR? (2) Is the present precision-recall ratio appropriate? (3) Between the grades and ranks of automatic evaluation, which correlates better with human evaluation? Five translator trainees majored respectively in Arabic, German, English, Japanese, and Spanish (a total of 25 students), translated four texts into Korean (a total of 100 texts). The translations were evaluated by two professional translators in each language and their evaluation results were compared with the outcome of the automatic evaluation. The results showed that the METEOR, recall and ranks correlated with the human ratings better than the BLEU, precision and scores. This and other findings from this experiment suggest that with the minimum number of ca. 12 translations, METEOR can be used at least when determining the order of student performance.

KEYWORDS

automatic evaluation, translation quality, validity, reliability, BLEU

Citation status

* References for papers published after 2023 are currently being built.

[confproc] 김유섭 / 2017 / 기계 번역에서의 평가 / 한국외대 통번역연구소 제18회 국제학술대회 <디지털 시대의 통번역> 발표집 : 337~357

[journal] 김진아 / 2002 / A Study on Quality Assessment Criteria in English-Korean Translation / 외국문학연구 / 외국문학연구소 (11) : 4~124

[book] 박혜주 / 2007 / 문학번역 평가 시스템 연구 / 한국문학번역원

[journal] 이영훈 / 2010 / Development of a translation assessment system for the study of Korean translations of French Great Novels / 통역과 번역 / 한국통역번역학회 12(2) : 149~180

[journal] 조성원 / 2007 / Rethinking ‘Faithfulness’ and ‘Readability’ in Translation Criticism / 안과밖:영미문학연구 / 영미문학연구회 (23) : 96~119

[confproc] 한국외대 번역평가인증 연구팀 / 2016 / 번역인증제도 (실무편) / 한국외대 통번역연구소 학술대회 <언어, 통번역의 평가 및 인증> 발표집 : 23~33

[confproc] Banerjee, Satanjeev / 2005 / METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments / Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization : 65~72

[journal] Chung, Hye-Yeon / 2020 / Automatische Evaluation der Humanübersetzung: BLEU vs. METEOR / Lebende Sprachen 65(1)

[confproc] Denkowski, Michael / 2011 / Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems / Proceedings of the 6th Workshop on Statistical Machine Translation : 85~91

[confproc] Han, Aaron L / 2012 / LEPOR: A Robust Evaluation Metric for Machine Translation with Augmented Factors / Proceedings of COLING 2012: Posters : 441~450

[book] House, Juliane / 1997 / Translation Quality Assessment: A Model Revisited / Tübingen

[book] Jakobsen, Arnt Lykke / 2011 / Methods and Strategies of Process research: Integrative Approaches in Translation Studies / John Benjamins : 37~55

[journal] Kunilovskaya, Maria / 2015 / How far do we agree on the quality of translation? / English Studies at NBU 1(1) : 18~31

[journal] Lai, Tzu-Yun / 2011 / Reliability and Validity of a Scale-based Assessment for Translation Tests / Meta 56(3) : 713~722

[web] Lavie, Alon / 2004 / The Significance of Recall in Automatic Metrics for MT Evaluation / https://www.cs.cmu.edu/~alavie/papers/Recall-AMTA-04.pdf

[confproc] PACTE / 2008 / Results of the Validation of the PACTE Translation Competence Model. Decision Making / AILA World Congress 2008. Multilingualism, Challenges & Opportunities : 1~5

[confproc] Papineni, Kishore / 2002 / BLEU: a Method for Automatic Evaluation of Machine Translation / Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), : 311~318

[web] Sasaki, Yutaka / 2007 / The truth of the F-measure / https://www.cs.odu.edu/~mukka/cs795sum10dm/Lecturenotes/Day3/F-measure-YS-26Oct07.pdf

[journal] Waddington, Christopher / 2001 / Should translations be assessed holistically or through error analysis? Hermesi / Journal of Linguistics 26 : 15~37

[journal] Waddington, Christopher / 2001 / Different Methods of Evaluating Student Translations: The Question of Validity / Meta 46(2) : 311~325

[journal] Williams, Malcolm / 2001 / The Application of Argumentation Theory to Translation Quality Assessment / Meta 46(2) : 326~344

KJCKorea
Journal Central

The Journal of Translation Studies 2023 KCI Impact Factor : 1.49

Application of Automatic Evaluation to Human Translation

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

The Journal of Translation Studies 2023 KCI Impact Factor : 1.49

Application of Automatic Evaluation to Human Translation

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (11)

REFERENCES (21) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.