Text Classification Using Heterogeneous Knowledge Distillation (이질적 지식증류를 활용한 텍스트 분류 방법론)

Yerin Yu (유예린); Namgyu Kim (김남규)

doi:10.9708/jksci.2022.27.10.029

Text Classification Using Heterogeneous Knowledge Distillation

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2022, 27(10), pp.29-41
DOI : 10.9708/jksci.2022.27.10.029
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : September 22, 2022
Accepted : October 24, 2022
Published : October 31, 2022

Yerin Yu ¹, Namgyu Kim ¹

¹국민대학교

Accredited

ABSTRACT

Recently, with the development of deep learning technology, a variety of huge models with excellent performance have been devised by pre-training massive amounts of text data. However, in order for such a model to be applied to real-life services, the inference speed must be fast and the amount of computation must be low, so the technology for model compression is attracting attention. Knowledge distillation, a representative model compression, is attracting attention as it can be used in a variety of ways as a method of transferring the knowledge already learned by the teacher model to a relatively small-sized student model. However, knowledge distillation has a limitation in that it is difficult to solve problems with low similarity to previously learned data because only knowledge necessary for solving a given problem is learned in a teacher model and knowledge distillation to a student model is performed from the same point of view. Therefore, we propose a heterogeneous knowledge distillation method in which the teacher model learns a higher-level concept rather than the knowledge required for the task that the student model needs to solve, and the teacher model distills this knowledge to the student model. In addition, through classification experiments on about 18,000 documents, we confirmed that the heterogeneous knowledge distillation method showed superior performance in all aspects of learning efficiency and accuracy compared to the traditional knowledge distillation.

KEYWORDS

Deep Learning, Knowledge Distillation, Text Classification, Model Compression

Citation status

* References for papers published after 2023 are currently being built.

[book] H. L. Erickson / 2017 / Concept-Based Curriculum and Instruction for the Thinking Classroom / Corwin

[other] G. Hinton / 2015 / Distilling the Knowledge in a Neural Network

[journal] J. Kim / 2018 / Paraphrasing Complex Network:Network Compression via Factor Transfer / Advances in neural information processing systems 31

[journal] J. Ba / 2014 / Do deep nets really need to be deep? / Advances in neural information processing systems 27

[confproc] S. I. Mirzadeh / 2020 / Improved Knowledge Distillation via Teacher Assistant / Proceedings of the AAAI conference on artificial intelligence 34(04) : 5191~5198

[other] A. Romero / 2015 / FITNETS: HINTS FOR THIN DEEP NETS

[other] Z. Huang / 2017 / Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

[confproc] B. Heo / 2019 / Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons / Proceedings of the AAAI Conference on Artificial Intelligence 33(01) : 3779~3787

[confproc] J. Yim / 2017 / A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 4133~4141

[other] S. Lee / 2019 / Graph-based Knowledge Distillation by Multi-head Attention Network

[confproc] Y. Liu / 2019 / Knowledge distillation via instance relationship graph / Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 7096~7104

[journal] J. Gou / 2021 / Knowledge Distillation: A Survey / International Journal of Computer Vision 129(6) : 1789~1819

[other] J. Devlin / 2019 / BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding

[other] S. Hahn / 2019 / Self-Knowledge Distillation in Natural Language Processing

[other] Y. C. Chen / 2020 / Distilling Knowledge Learned in BERT for Text Generation

[confproc] S. Arora / 2019 / On Knowledge Distillation from Complex Networks for Response Prediction / Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 : 3813~3822

[other] S. Shakeri / 2019 / Knowledge Distillation in Document Retrieval

[journal] S. Zhang / 2022 / Cross-domain knowledge distillation for text classification / Neurocomputing 509 : 11~202022

[journal] 김남규 / 2017 / Investigations on Techniques and Applications of Text Analytics / 한국통신학회논문지 / 한국통신학회 42(2) : 471~492

[journal] 김성훈 / 2021 / Recursive Oversampling Method for Improving Classification Performance of Class Unbalanced Data in Patent Document Automatic Classification / 전자공학회논문지 / 대한전자공학회 58(4) : 43~49

[confproc] B. Dipto / 2021 / Research Paper Classification Scheme based on Word Embedding / Proceedings of the Korea Information Processing Society Conference 28(2)

[confproc] H. Son / 2021 / Rule-based filtering and deep learning LSTM e-mail spam classification / Proceedings of the Korean Information Science Society Conference : 105~107

[journal] 지선미 / 2018 / A Twitter News-Classification Scheme Using Semantic Enrichment of Word Features / 정보과학회논문지 / 한국정보과학회 45(10) : 1045~1055

[journal] 윌리엄 / 2018 / Improving the Accuracy of Document Classification by Learning Heterogeneity / 지능정보연구 / 한국지능정보시스템학회 24(3) : 21~44

[journal] 박상언 / 2021 / Analysis of the Status of Natural Language Processing Technology Based on Deep Learning / 한국빅데이터학회 학회지 / 사)한국빅데이터학회 6(1) : 63~81

[other] T. Mikolov / 2013 / Efficient Estimation of Word Representations in Vector Space

[other] K. Clark / 2020 / ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

[other] V. Sanh / 2020 / DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

[other] X. Jiao / 2020 / TinyBERT: Distilling BERT for Natural Language Understanding

[confproc] K. Lang / 1995 / NewsWeeder: Learning to Filter Netnews / Proceedings of the Twelfth International Conference on Machine Learning : 331~339

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Text Classification Using Heterogeneous Knowledge Distillation

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Text Classification Using Heterogeneous Knowledge Distillation

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (30) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.