Fusion Machine Learning Strategies for Multi-modal Sensor-based Hand Gesture Recognition
Received: 14 March 2022 | Revised: 26 March 2022 | Accepted:1 April 2022 | Online: 6 June 2022
Corresponding author: H. G. Doan
Abstract
Hand gesture recognition has attracted the attention of many scientists, because of its high applicability in fields such as sign language expression and human machine interaction. Many approaches have been deployed to detect and recognize hand gestures, like wearable devices, image information, and/or a combination of sensors and computer vision. However, the method of using wearable sensors brings much higher accuracy and is less affected by occlusion, lighting conditions, and complex background. Existing solutions separately utilize sensor information and/or only use sensor information processing and decision-making algorithms over conventional threshold comparison algorithms and do not analyze data or utilize machine learning algorithms. In this paper, a multi-modal solution is proposed that combines information for measuring the curvature of the fingers and sensors for measuring angular velocity and acceleration. The provided information from the sensors is normalized and analyzed and various fusion strategies are used. Then, the most suitable algorithm for these sensor-based multiple modalities is proposed. The proposed system also analyzes the differences between gestures and actions that are almost similar but in fact, they are just normal moving gestures.
Keywords:
Hand Glove, Acceremory Sensor, Hand Gesture Recognition, Flex Sensor, Human-Machine InteractionDownloads
References
F. Aiolli, G. Da San Martino, and A. Sperduti, "A Kernel Method for the Optimization of the Margin Distribution," in Artificial Neural Networks - ICANN 2008, Berlin, Heidelberg, Germany, 2008, pp. 305–314. DOI: https://doi.org/10.1007/978-3-540-87536-9_32
T.-H. Tran, H.-N. Tran, and H.-G. Doan, "Dynamic Hand Gesture Recognition from Multi-modal Streams Using Deep Neural Network," in Multi-disciplinary Trends in Artificial Intelligence, 2019, pp. 156–167. DOI: https://doi.org/10.1007/978-3-030-33709-4_14
Dang-Manh Truong was with Hanoi University of Science Technology, Vietnam, D.-M. Truong, H.-G. Doan, T.-H. Tran, H. Vu, and T.-L. Le, "Robustness Analysis of 3D Convolutional Neural Network for Human Hand Gesture Recognition," International Journal of Machine Learning and Computing, vol. 9, no. 2, pp. 135–142, Apr. 2019. DOI: https://doi.org/10.18178/ijmlc.2019.9.2.777
P. Das, T. Ahmed, and Md. F. Ali, "Static Hand Gesture Recognition for American Sign Language using Deep Convolutional Neural Network," in 2020 IEEE Region 10 Symposium (TENSYMP), Dhaka, Bangladesh, Jun. 2020, pp. 1762–1765. DOI: https://doi.org/10.1109/TENSYMP50017.2020.9230772
J. Zamora-Mora and M. Chacón-Rivas, "Real-Time Hand Detection using Convolutional Neural Networks for Costa Rican Sign Language Recognition," in 2019 International Conference on Inclusive Technologies and Education (CONTIE), San Jose del Cabo, Mexico, Jul. 2019, pp. 180–1806. DOI: https://doi.org/10.1109/CONTIE49246.2019.00042
N. Sarhan and S. Frintrop, "Transfer Learning For Videos: From Action Recognition To Sign Language Recognition," in 2020 IEEE International Conference on Image Processing (ICIP), Jul. 2020, pp. 1811–1815. DOI: https://doi.org/10.1109/ICIP40778.2020.9191289
S. Veluchamy, L. R. Karlmarx, and J. J. Sudha, "Vision based gesturally controllable human computer interaction system," in 2015 International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM), Avadi, India, Feb. 2015, pp. 8–15. DOI: https://doi.org/10.1109/ICSTM.2015.7225383
H.-G. Doan, H. Vu, T.-H. Tran, and E. Castelli, "Improvements of RGB-D hand posture recognition using an user-guide scheme," in 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), Siem Reap, Cambodia, Jul. 2015, pp. 24–29. DOI: https://doi.org/10.1109/ICCIS.2015.7274542
S. Lee, K. Park, J. Lee, and K. Kim, "User Study of VR Basic Controller and Data Glove as Hand Gesture Inputs in VR Games," in 2017 International Symposium on Ubiquitous Virtual Reality (ISUVR), Nara, Japan, Jun. 2017. DOI: https://doi.org/10.1109/ISUVR.2017.16
N. E. A. Rashid, Y. A. I. M. Nor, K. K. M. Sharif, Z. I. Khan, and N. A. Zakaria, "Hand Gesture Recognition using Continuous Wave (CW) Radar based on Hybrid PCA-KNN," in 2021 IEEE Symposium on Wireless Technology Applications (ISWTA), Shah Alam, Malaysia, Dec. 2021, pp. 88–92. DOI: https://doi.org/10.1109/ISWTA52208.2021.9587404
G. Zhang, S. Lan, K. Zhang, and L. Ye, "Temporal-Range-Doppler Features Interpretation and Recognition of Hand Gestures Using mmW FMCW Radar Sensors," in 2020 14th European Conference on Antennas and Propagation (EuCAP), Copenhagen, Denmark, Mar. 2020. DOI: https://doi.org/10.23919/EuCAP48036.2020.9135694
P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz, "Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 4207–4215. DOI: https://doi.org/10.1109/CVPR.2016.456
J. J. Raval and R. Gajjar, "Real-time Sign Language Recognition using Computer Vision," in 2021 3rd International Conference on Signal Processing and Communication (ICPSC), Coimbatore, India, Feb. 2021, pp. 542–546. DOI: https://doi.org/10.1109/ICSPC51351.2021.9451709
J. Fink, B. Frénay, L. Meurant, and A. Cleve, "LSFB-CONT and LSFB-ISOL: Two New Datasets for Vision-Based Sign Language Recognition," in 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, Jul. 2021. DOI: https://doi.org/10.1109/IJCNN52387.2021.9534336
X. Chu, J. Liu, and S. Shimamoto, "A Sensor-Based Hand Gesture Recognition System for Japanese Sign Language," in 2021 IEEE 3rd Global Conference on Life Sciences and Technologies (LifeTech), Nara, Japan, Mar. 2021, pp. 311–312. DOI: https://doi.org/10.1109/LifeTech52111.2021.9391981
K. Liu, C. Chen, R. Jafari, and N. Kehtarnavaz, "Multi-HMM classification for hand gesture recognition using two differing modality sensors," in 2014 IEEE Dallas Circuits and Systems Conference (DCAS), Richardson, TX, USA, Jul. 2014. DOI: https://doi.org/10.1109/DCAS.2014.6965338
S. Zhu, A. Stuttaford-Fowler, A. Fahmy, C. Li, and J. Sienz, "Development of a Low-cost Data Glove using Flex Sensors for the Robot Hand Teleoperation," in 2021 3rd International Symposium on Robotics Intelligent Manufacturing Technology (ISRIMT), Changzhou, China, Sep. 2021, pp. 47–51. DOI: https://doi.org/10.1109/ISRIMT53730.2021.9596972
A. K. Panda, R. Chakravarty, and S. Moulik, "Hand Gesture Recognition using Flex Sensor and Machine Learning Algorithms," in 2020 IEEE-EMBS Conference on Biomedical Engineering and Sciences (IECBES), Langkawi Island, Malaysia, Mar. 2021, pp. 449–453. DOI: https://doi.org/10.1109/IECBES48179.2021.9398789
S. S. Kumar, R. Gatti, S. K. N. Kumar, N. Nataraja, R. P. Prasad, and T. Sarala, "Glove Based Deaf-Dumb Sign Language Interpreter," in 2021 International Conference on Recent Trends on Electronics, Information, Communication Technology (RTEICT), Bangalore, India, Dec. 2021, pp. 947–950. DOI: https://doi.org/10.1109/RTEICT52294.2021.9573990
Z. Zou, Q. Wu, Y. Zhang, and K. Wen, "Design of Smart Car Control System for Gesture Recognition Based on Arduino," in 2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), Guangzhou, China, Jan. 2021, pp. 695–699. DOI: https://doi.org/10.1109/ICCECE51280.2021.9342137
X. Chen et al., "A Wearable Hand Rehabilitation System With Soft Gloves," IEEE Transactions on Industrial Informatics, vol. 17, no. 2, pp. 943–952, Oct. 2021. DOI: https://doi.org/10.1109/TII.2020.3010369
S. Mugala, P. Jjagwe, and J. Asiimwe, "Glove Based Sign Interpreter for Medical Emergencies," in 2019 IST-Africa Week Conference (IST-Africa), Nairobi, Kenya, Feb. 2019. DOI: https://doi.org/10.23919/ISTAFRICA.2019.8764816
P. Telluri, S. Manam, S. Somarouthu, J. M. Oli, and C. Ramesh, "Low cost flex powered gesture detection system and its applications," in 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India, Jul. 2020, pp. 1128–1131. DOI: https://doi.org/10.1109/ICIRCA48905.2020.9182833
H. V. T. Chi, D. L. Anh, N. L. Thanh, and D. Dinh, "English-Vietnamese Cross-Lingual Paraphrase Identification Using MT-DNN," Engineering, Technology & Applied Science Research, vol. 11, no. 5, pp. 7598–7604, Oct. 2021. DOI: https://doi.org/10.48084/etasr.4300
N. T. T. Vu, N. P. Tran, and N. H. Nguyen, "Recurrent Neural Network-based Path Planning for an Excavator Arm under Varying Environment," Engineering, Technology & Applied Science Research, vol. 11, no. 3, pp. 7088–7093, Jun. 2021. DOI: https://doi.org/10.48084/etasr.4125
M. Wattenberg, F. Viégas, and I. Johnson, "How to Use t-SNE Effectively," Distill, vol. 1, no. 10, Oct. 2016, Art. no. e2. DOI: https://doi.org/10.23915/distill.00002
A. A. Alzamil, "Assessment of Uplink Massive MIMO in Scattering Environment," Engineering, Technology & Applied Science Research, vol. 10, no. 5, pp. 6290–6293, Oct. 2020. DOI: https://doi.org/10.48084/etasr.3743
O. Luzanin and M. Plancak, "Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network," Assembly Automation, vol. 34, no. 1, pp. 94–105, Jan. 2014. DOI: https://doi.org/10.1108/AA-03-2013-020
Downloads
How to Cite
License
Copyright (c) 2022 H. G. Doan, N. T. Nguyen
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
