Knowledge Distillation based-on Internal/External Correlation Learning (모델 내부/외부 특징량 상관 학습을 통한 지식 증류)

Hun-Beom Bak (박훈범); Seung-Hwan Bae (배승환)

doi:10.9708/jksci.2023.28.04.031

Knowledge Distillation based-on Internal/External Correlation Learning

Journal of The Korea Society of Computer and Information
Abbr : JKSCI
2023, 28(4), pp.31-39
DOI : 10.9708/jksci.2023.28.04.031
Publisher : The Korean Society Of Computer And Information
Research Area : Engineering > Computer Science
Received : March 7, 2023
Accepted : April 3, 2023
Published : April 28, 2023

Hun-Beom Bak ¹, Seung-Hwan Bae ¹

¹인하대학교

Accredited

ABSTRACT

In this paper, we propose an Internal/External Knowledge Distillation (IEKD), which utilizes both external correlations between feature maps of heterogeneous models and internal correlations between feature maps of the same model for transferring knowledge from a teacher model to a student model. To achieve this, we transform feature maps into a sequence format and extract new feature maps suitable for knowledge distillation by considering internal and external correlations through a transformer. We can learn both internal and external correlations by distilling the extracted feature maps and improve the accuracy of the student model by utilizing the extracted feature maps with feature matching. To demonstrate the effectiveness of our proposed knowledge distillation method, we achieved 76.23% Top-1 image classification accuracy on the CIFAR-100 dataset with the “ResNet-32×4/VGG-8” teacher and student combination and outperformed the state-of-the-art KD methods.

KEYWORDS

Knowledge distillation, model compression, transformer, correlation learning, Image classification

Citation status

* References for papers published after 2023 are currently being built.

[confproc] He, Kaiming / 2016 / Deep residual learning for image recognition / Proceedings of the IEEE conference on computer vision and pattern recognition : 770~778

[journal] Shaoqing Ren / 2017 / Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks / IEEE Transactions on Pattern Analysis and Machine Intelligence / Institute of Electrical and Electronics Engineers (IEEE) 39(6) : 1137~1149

[book] Han, Song / 2016 / Deep compression:Compressing deep neural networks with pruning, trained quantization and huffman coding / International Conference on Learning Representations

[confproc] Mohammad Rastegari / 2016 / Xnor-net: Imagenet classification using binary convolutional neural networks / Computer Vision–ECCV 2016:14th European Conference : 525~542

[web] G. Hinton / 2015 / Distilling the Knowledge in a Neural Network / arXiv / DOI:10.48550/arXiv.1503.02531

[confproc] A. Romero / 2015 / Fitnets: Hints for thin deep nets / International Conference on Learning Representations

[confproc] Chen, Defang / 2021 / Cross-layer distillation with semantic calibration / Proceedings of the AAAI Conference on Artificial Intelligence 35(8) : 7028~7036

[journal] Hao-Ting Li / 2019 / Layer-Level Knowledge Distillation for Deep Neural Network Learning / Applied Sciences / MDPI AG 9(10) : 1966~

[confproc] Han, Kai / 2020 / Ghostnet: More features from cheap operations / Proceedings of the IEEE/CVF conference on computer vision and pattern recognition : 1577~1586

[confproc] Liu, Li / 2021 / Exploring inter-channel correlation for diversity-preserved knowledge distillation / Proceedings of the IEEE/CVF International Conference on Computer Vision : 8251~8260

[other] Vaswani, Ashish / 2017 / Attention is all you need / Advances in neural information processing systems 30 : 6000~6010

[report] Alex Krizhevsky / 2009 / Learning multiple layers of features from tiny images : 1~60

[confproc] Yang, Zhendong / 2022 / Focal and global knowledge distillation for detectors / Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 4633~4642

[web] Gong, Yuan / 2022 / Cmkd: Cnn/transformer-based cross-model knowledge distillation for audio classification / arXiv / arXiv:2203.06760

[confproc] Zhao, Borui / 2022 / Decoupled knowledge distillation / Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition : 11943~11952

[confproc] Chen, Defang / 2022 / Knowledge distillation with the reused teacher classifier / Proceedings of the IEEE/CVF conference on computer vision and pattern recognition : 11923~11932

[confproc] Beyer, Lucas / 2022 / Knowledge distillation: A good teacher is patient and consistent / Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 10915~10924

[confproc] Park, Wonpyo / 2019 / Relational knowledge distillation / Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 3962~3971

[confproc] S. Zagoruyko / 2017 / Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer / International Conference on Learning Representations

[confproc] F. Tung / 2019 / Similarity-preserving knowledge distillation / International Conference on Computer Vision : 1365~1374

[confproc] S. Ahn / 2019 / Variational information distillation for knowledge transfer / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 9163~9171

[confproc] N. Passalis / 2020 / Heterogeneous knowledge distillation using information flow modeling / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 2336~2345

[confproc] Lin, Sihao / 2022 / Knowledge distillation via the target-aware transformer / Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition : 10905~10914

[other] Paszke, Adam / 2019 / Pytorch: An imperative style, high-performance deep learning library / Advances in neural information processing systems 32 : 8026~8037

[confproc] Karen Simonyan / 2015 / Very deep convolutional networks for large-scale image recognition / International Conference on Learning Representations

[confproc] Sandler, Mark / 2018 / Mobilenetv2: Inverted residuals and linear bottlenecks / Proceedings of the IEEE conference on computer vision and pattern recognition : 4510~4520

[confproc] Zagoruyko, Sergey / 2016 / Wide residual networks / Proceedings of the British Machine Vision Conference : 87~81

[confproc] Ma, Ningning / 2018 / Shufflenet v2: Practical guidelines for efficient cnn architecture design / Proceedings of the European conference on computer vision (ECCV) : 116~131

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Knowledge Distillation based-on Internal/External Correlation Learning

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of The Korea Society of Computer and Information 2023 KCI Impact Factor : 0.65

Knowledge Distillation based-on Internal/External Correlation Learning

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (28) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.