Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh (MediaPipe Face Mesh를 이용한 얼굴 제스처 기반의 사용자 인터페이스의 성능 개선)

JinWang Mok (목진왕); Noyoon Kwak (곽노윤)

doi:10.20465/KIOTS.2023.9.6.125

Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh

Journal of Internet of Things and Convergence
Abbr : JKIOTS
2023, 9(6), pp.125-134
DOI : 10.20465/KIOTS.2023.9.6.125
Publisher : The Korea Internet of Things Society
Research Area : Engineering > Computer Science > Internet Information Processing
Received : September 29, 2023
Accepted : November 23, 2023
Published : December 29, 2023

JinWang Mok ¹, Noyoon Kwak ²

¹광주과학기술원
²백석대학교

Accredited

ABSTRACT

The purpose of this paper is to propose a method to improve the performance of the previous research is characterized by recognizing facial gestures from the 3D coordinates of seven landmarks selected from the MediaPipe Face Mesh model, generating corresponding user events, and executing corresponding commands. The proposed method applied adaptive moving average processing to the cursor positions in the process to stabilize the cursor by alleviating microtremor, and improved performance by blocking temporary opening/closing discrepancies between both eyes when opening and closing both eyes simultaneously. As a result of the usability evaluation of the proposed facial gesture interface, it was confirmed that the average recognition rate of facial gestures was increased to 98.7% compared to 95.8% in the previous research.

KEYWORDS

Facial Gesture Recognition, Binocular Opening and Closing, MediaPipe, Face Mesh Model, NUI

Citation status

* References for papers published after 2023 are currently being built.

[journal] F. Karray / 2008 / Human-Computer Interaction : Overview on State of the Art / International Journal on Smart Sensing and Intelligent Systems 1(1) : 137~159

[journal] T. H. Tsai / 2020 / Design of Hand Gesture Recognition System for Human-computer Interaction / Multimedia Tools and Applications 79(9-10) : 5989~6007

[journal] 김규민 / 2019 / Real-Time Hand Gesture Recognition Based on Deep Learning / 멀티미디어학회논문지 / 한국멀티미디어학회 22(4) : 424~431

[book] B. Kumar / 2023 / Handbook of Research on AI-Based Technologies and Applications in the Era of the Metaverse : 542~358

[journal] Q. Gao / 2021 / Dynamic Hand Gesture Recognition Based on 3D Hand Pose Estimation for Human-robot Interaction / IEEE Sensors Journal : 17421~17430

[confproc] H. Kaur / 2016 / A Review : Study of Various Techniques of Hand Gesture Recognition / Proceedings of 2016 IEEE 1st International Conference on Power Electronics, Intelligent Control and Energy Systems : 1~5

[journal] Y. Li / 2019 / Gesture Interaction in Virtual Reality / Virtual Reality and Intelligent Hardware : 84~112

[confproc] C. A. Cruz / 2023 / Sequential Eyelid Gestures for User Interfaces in VR / Proceedings of 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops

[confproc] A. Shimada / 2013 / Hand Gesture Based TV Control System— Towards Both User-Machine-friendly Gesture Applications / Proceedings of The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision : 121~126

[journal] H. Stern / 2006 / A Real-Time Hand Gesture Interface for Medical Visualization Applications / Applications of Soft Computing / Springer 36 : 153~162

[confproc] G. Pala / 2021 / Machine Learning-based Hand Sign Recognition / Proceedings of 2021 International Conference on Artificial Intelligence and Smart Systems : 356~363

[confproc] M. Iskandar / 2023 / Artificial Intelligence-based Human Gesture Tracking Control Techniques of Tello EDU Quadrotor Drone / Proceedings of International Conference on Green Energy, Computing and Intelligent Technology

[web] / MANO / https://mano.is.tue.mpg.de

[confproc] N. Qian / 2020 / HTML : A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalization / Proceedings of the European Conference on Computer Vision : 54~71

[journal] S. An / 2022 / Fast Hand: Fast Monocular Hand Pose Estimation on Embedded Systems / Journal of Systems Architecture 122

[confproc] Z. Fan / 2021 / Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition : 1~10

[journal] 장명수 / 2019 / Implementation of User Gesture Recognition System for manipulating a Floating Hologram Character / 한국인터넷방송통신학회 논문지 / 한국인터넷방송통신학회 19(2) : 143~149

[confproc] L. Chen / 2021 / MVHM : A Large-Scale multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation / Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition : 836~845

[web] / Google MediaPipe / https://developers.google.com/mediapipe

[journal] 허경용 / 2021 / Hierarchical Hand Pose Model for Hand Expression Recognition / 한국정보통신학회논문지 / 한국정보통신학회 25(10) : 1323~1329

[journal] 허경용 / 2021 / Hand Expression Recognition for Virtual Blackboard / 한국정보통신학회논문지 / 한국정보통신학회 25(12) : 1770~1776

[journal] 송복득 / 2022 / Design and Implementation of a Stereoscopic Image Control System based on User Hand Gesture Recognition / 한국정보통신학회논문지 / 한국정보통신학회 26(3) : 396~402

[journal] 송락빈 / 2023 / User Interface Using Hand Gesture Recognition Based on MediaPipe Hands Model / 멀티미디어학회논문지 / 한국멀티미디어학회 26(2) : 103~115

[journal] J. Prameela / 2022 / Mouse Handling Using Facial Gesture / International Research Journal of Modernization in Engineering Technology and Science 04(5) : 468~475

[journal] S. Sreeni / 2023 / Mouse Cursor Control Using Facial Movements-An HCI Application / International Journal of Techno-Engineering 15(2) : 270~274

[confproc] Z. Sharifisoraki / 2023 / A Novel Face Recognition Using Specific Values from Deep Neural Network-based Landmarks / Proceedings of 2023 IEEE International Conference on Consumer Electronics

[thesis] S. Thino / 2023 / Developing a Program to Detect Face Direction and the State of Partially Closed Eyes / Master / Naresuan University

[confproc] J. Mok / 2023 / Facial Gesture-based User Interface Using MediaPipe Face Mesh / Proceedings of 2023 Summer Annual Conference of The Institute of Electronics and Information Engineers : 1407~1411

This paper was written with support from the National Research Foundation of Korea.

KJCKorea
Journal Central

Journal of Internet of Things and Convergence 2023 KCI Impact Factor : 0.89

Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh

ABSTRACT

KEYWORDS

Citation status

* References for papers published after 2023 are currently being built.

Journal of Internet of Things and Convergence 2023 KCI Impact Factor : 0.89

Performance Improvement of Facial Gesture-based User Interface Using MediaPipe Face Mesh

ABSTRACT

KEYWORDS

Statistics

Tools

Issue List

Citation status

KCI Citation Counts (0)

REFERENCES (28) * References for papers published after 2023 are currently being built.

Search PDF

Citation

* References for papers published after 2023 are currently being built.