HTML Tag Depth Embedding: An Input Embedding Method of the BERT Model for Improving Web Document Reading Comprehension Performance (HTML 태그 깊이 임베딩: 웹 문서 기계 독해 성능 개선을 위한 BERT 모델의 입력 임베딩 기법)

Jin-Wang Mok (목진왕); Jang Hyun Jae (장현재); Hyun-Seob Lee (이현섭)

doi:10.20465/KIOTS.2022.8.5.017

HTML Tag Depth Embedding: An Input Embedding Method of the BERT Model for Improving Web Document Reading Comprehension Performance

Journal of Internet of Things and Convergence
Abbr : JKIOTS
2022, 8(5), pp.17-25
DOI : 10.20465/KIOTS.2022.8.5.017
Publisher : The Korea Internet of Things Society
Research Area : Engineering > Computer Science > Internet Information Processing
Received : July 18, 2022
Accepted : September 6, 2022
Published : October 31, 2022

Jin-Wang Mok ¹, Jang Hyun Jae ¹, Hyun-Seob Lee ¹

¹백석대학교

Accredited

ABSTRACT

Recently the massive amount of data has been generated because of the number of edge devices increases. And especially, the number of raw unstructured HTML documents has been increased. Therefore, MRC(Machine Reading Comprehension) in which a natural language processing model finds the important information within an HTML document is becoming more important. In this paper, we propose HTDE(HTML Tag Depth Embedding Method), which allows the BERT to train the depth of the HTML document structure. HTDE makes a tag stack from the HTML document for each input token in the BERT and then extracts the depth information. After that, we add a HTML embedding layer that takes the depth of the token as input to the step of input embedding of BERT. Since tokenization using HTDE identifies the HTML document structures through the relationship of surrounding tokens, HTDE improves the accuracy of BERT for HTML documents. Finally, we demonstrated that the proposed idea showing the higher accuracy compared than the accuracy using the conventional embedding of BERT.

KEYWORDS

Natural Language Processing, Machine Reading Comprehension, Embeddings, HTML, BERT

Citation status

* References for papers published after 2023 are currently being built.

[journal] 이진 / 2020 / Analysis of the Hardware Structures of the IoT Device Platforms for the Minimal Power Consumption / 사물인터넷융복합논문지 / 한국사물인터넷학회 6(2) : 11~18

[journal] 이현섭 / 2022 / A Prediction-Based Data Read Ahead Policy using Decision Tree for improving the performance of NAND flash memory based storage devices / 사물인터넷융복합논문지 / 한국사물인터넷학회 8(4) : 9~15

[journal] 이성훈 / 2020 / A Software Engineering-Based Software Development Progress Analysis in IoT Environment / 사물인터넷융복합논문지 / 한국사물인터넷학회 6(2) : 87~92

[journal] T.Kwiatkowski / 2019 / Natural questions: a benchmark for question answering research / Transactions of the Association for Computational Linguistics 7 : 453~466

[confproc] D. Chen / 2017 / Reading wikipedia to answer open-domain questions / Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics 1 : 1870~1879

[confproc] E.Choi / 2018 / QuAC: Question answering in context / Proceedings of the 2018Conference on Empirical Methods in Natural Language Processing : 2174~2184

[journal] A.Asai / 2021 / One question answering model for many languages with cross-lingual dense passage retrieval / Advances in Neural Information Processing Systems 34 : 7547~7560

[confproc] J.Devlin / 2019 / Bert:Pre-training of deep bidirectional transformers for language understanding / Proceedings of the 2019Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 : 4171~4186

[confproc] B.Wang / 2020 / On position embeddings in bert / International Conference on Learning Representations

[confproc] P.Rajpurkar / 2016 / Squad:100,000+ questions for machine comprehension of text / Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing : 2383~2392

[confproc] Z.Yang / 2018 / HotpotQA: A dataset for diverse, explainable multi-hop question answering / Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing : 2369~2380

[confproc] S.Y.Lim / 2018 / Korquad: Korean qa dataset for machine comprehension / Proceeding of the Conference of the Korea Information Science Society 45(2) : 539~541

[journal] 김영민 / 2020 / KorQuAD 2.0: Korean QA Dataset for Web Document Machine Comprehension / 정보과학회논문지 / 한국정보과학회 47(6) : 577~586

[confproc] I.Sutskever / 2014 / Sequence to sequence learning with neural networks / Proceedings of the 27th International Conference on Neural Information Processing Systems 2 : 3104~3112

[confproc] D.Bahdanau / 2015 / Neural machine translation by jointly learning to align and translate / 3rd International Conference on Learning Representations

[journal] A.Vaswani / 2017 / Attention is all you need / Advances in neural information processing systems 30 : 5998~6008

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of Internet of Things and Convergence 2023 KCI Impact Factor : 0.89

HTML Tag Depth Embedding: An Input Embedding Method of the BERT Model for Improving Web Document Reading Comprehension Performance

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of Internet of Things and Convergence 2023 KCI Impact Factor : 0.89

HTML Tag Depth Embedding: An Input Embedding Method of the BERT Model for Improving Web Document Reading Comprehension Performance

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (1)

REFERENCES (16) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.