Clinical Hand Gesture Recognition in Intubated ICU Patients Using CNN-LSTM under Leave-One-Subject-Out Evaluation

Emy Setyaningsih; Erma Susanti; Septiana Fathonah; Taukhit

doi:10.48084/etasr.19302

Authors

Emy Setyaningsih Department of Computer Systems Engineering, Akprind University, Indonesia
Erma Susanti Department of Informatics, Akprind University, Indonesia
Septiana Fathonah Department of Emergency and Critical Care Nursing, Notokusumo School of Health Sciences, Yogyakarta, Indonesia
Taukhit Department of Nursing Management, Notokusumo School of Health Sciences, Yogyakarta, Indonesia

Volume: 16 | Issue: 4 | Pages: 37523-37530 | August 2026 | https://doi.org/10.48084/etasr.19302

Received: 13 April 2026 | Revised: 19 May 2026 | Accepted: 3 June 2026 | Online: 11 June 2026

Corresponding author: Emy Setyaningsih

Abstract

Non-verbal communication remains challenging for conscious intubated ICU patients who are unable to speak. This study investigates whether bedside hand gestures can be recognized from real ICU video recordings using a spatio-temporal deep learning framework. A private dataset comprising 20 videos from 10 intubated patients was organized into five clinically relevant gesture classes: Agreement, Discomfort, Neutral, RequestHelp, and Suction. Four ImageNet-pretrained CNN backbones, namely DenseNet121, ResNet50, MobileNetV2, and EfficientNetB0, were integrated with an LSTM layer to capture temporal gesture dynamics. Model performance was assessed under strict Leave-One-Subject-Out (LOSO) evaluation to measure generalization to unseen patients. Among the models evaluated, EfficientNetB0–LSTM achieved the best overall performance, with an accuracy of 0.40 and a micro-AUC of 0.5669. However, class-wise discrimination remained uneven, and all models showed limited sensitivity to minority and visually subtle gesture classes, with predictions frequently biased toward RequestHelp. These findings indicate that hand gesture recognition from real bedside ICU videos is considerably more challenging than recognition in controlled settings and provide an initial subject-independent benchmark for gesture-based communication support in critical care.

Keywords:

clinical hand gesture recognition, CNN-LSTM, intubated ICU patients, leave-one-subject-out evaluation, non-verbal communication

References

[1] A. Anand, R. Panda, S. Kodamanchili, T. Gowthaman, R. R. Nair, and K. K. Bhardwaj, "Communication with Patients on Mechanical Ventilation: A Review of Existing Technologies," Indian Journal of Critical Care Medicine, vol. 26, no. 6, pp. 756–757, June 2022.

[2] S. Brambilla, D. Ausili, G. Locatelli, S. Di Mauro, G. Bellani, and M. Luciani, "Communication difficulties in mechanically ventilated voiceless patients in intensive care units: A qualitative study," Nursing in Critical Care, vol. 30, no. 3, May 2025, Art. no. e70037.

[3] M. Kyranou, C. Cheta, and E. Pampoulou, "Communicating with mechanically ventilated patients who are awake. A qualitative study on the experience of critical care nurses in Cyprus during the COVID-19 pandemic," PLOS ONE, vol. 17, no. 12, Dec. 2022, Art. no. e0278195.

[4] C. Çelebi and K. Ö. Yeşilyurt, "Ensuring Effective Communication with Patients Receiving Mechanical Ventilation Support in Intensive Care Units: Current Communication Materials," Cyprus Journal of Medical Sciences, Aug. 2025.

[5] N. R. Kuruppu, W. Chaboyer, A. Abayadeera, and K. Ranse, "Augmentative and alternative communication tools for mechanically ventilated patients in intensive care units: A scoping review," Australian Critical Care, vol. 36, no. 6, pp. 1095–1109, Nov. 2023.

[6] M. LaValley, T. Chavers-Edgar, M. Wu, R. Schlosser, and R. Koul, "Augmentative and Alternative Communication Interventions in Critical and Acute Care With Mechanically Ventilated and Tracheostomy Patients: A Scoping Review," American Journal of Speech-Language Pathology, vol. 33, no. 5, pp. 2667–2686, Sept. 2024.

[7] K. Aurangzeb, K. Javeed, M. Alhussein, I. Rida, S. I. Haider, and A. Parashar, "Deep Learning Approach for Hand Gesture Recognition: Applications in Deaf Communication and Healthcare," Computers, Materials & Continua, vol. 78, no. 1, pp. 127–144, 2024.

[8] H. P. J. Dutta, M. K. Bhuyan, D. R. Neog, K. F. MacDorman, and R. H. Laskar, "Patient Assistance System Based on Hand Gesture Recognition," IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1–13, 2023.

[9] L. Alzubaidi et al., "Review of deep learning: concepts, CNN architectures, challenges, applications, future directions," Journal of Big Data, vol. 8, no. 1, Mar. 2021, Art. no. 53.

[10] M. Linardakis, I. Varlamis, and G. T. Papadopoulos, "Survey on Hand Gesture Recognition From Visual Input," IEEE Access, vol. 13, pp. 135373–135406, 2025.

[11] M. Oudah, A. Al-Naji, and J. Chahl, "Hand Gesture Recognition Based on Computer Vision: A Review of Techniques," Journal of Imaging, vol. 6, no. 8, July 2020, Art. no. 73.

[12] R. Tripathi and B. Verma, "Survey on vision-based dynamic hand gesture recognition," The Visual Computer, vol. 40, no. 9, pp. 6171–6199, Sept. 2024.

[13] L. I. Barona López, F. M. Ferri, J. Zea, Á. L. Valdivieso Caraguay, and M. E. Benalcázar, "CNN-LSTM and post-processing for EMG-based hand gesture recognition," Intelligent Systems with Applications, vol. 22, June 2024, Art. no. 200352.

[14] R. E. Nogales and M. E. Benalcázar, "Hand Gesture Recognition Using Automatic Feature Extraction and Deep Learning Algorithms with Memory," Big Data and Cognitive Computing, vol. 7, no. 2, May 2023, Art. no. 102.

[15] M. Ur Rehman et al., "Dynamic Hand Gesture Recognition Using 3D-CNN and LSTM Networks," Computers, Materials & Continua, vol. 70, no. 3, pp. 4675–4690, 2022.

[16] H. G. Doan and N. T. Nguyen, "Fusion Machine Learning Strategies for Multi-modal Sensor-based Hand Gesture Recognition," Engineering, Technology & Applied Science Research, vol. 12, no. 3, pp. 8628–8633, June 2022.

[17] A. O. Hashi, S. Z. M. Hashim, and A. B. Asamah, "Dynamic Adaptation in Deep Learning for Enhanced Hand Gesture Recognition," Engineering, Technology & Applied Science Research, vol. 14, no. 4, pp. 15836–15841, Aug. 2024.

[18] P. Bansal, T. Mishra, H. Solanki, and S. Patil, "Hand Gesture Recognition Using Machine Learning Technique," in 2024 IEEE International Conference on Computing, Power and Communication Technologies (IC2PCT), Feb. 2024, pp. 322–326.

[19] T. L. Dang, S. D. Tran, T. H. Nguyen, S. Kim, and N. Monet, "An improved hand gesture recognition system using keypoints and hand bounding boxes," Array, vol. 16, Dec. 2022, Art. no. 100251.

[20] H. H. Li and C. C. Hsieh, "Dynamic Hand Gesture Recognition Using MediaPipe and Transformer," in IEEE ICEIB 2025, Sept. 2025, Art. no. 22.

[21] C. N. Rang, P. Jerónimo, C. Mora, and S. Jardim, "Hand Gesture Recognition using Machine Learning," Procedia Computer Science, vol. 256, pp. 198–205, 2025.

[22] A. Sharma, A. Mittal, S. Singh, and V. Awatramani, "Hand Gesture Recognition using Image Processing and Feature Extraction Techniques," Procedia Computer Science, vol. 173, pp. 181–190, 2020.

[23] N. Al Mudawi et al., "Innovative healthcare solutions: robust hand gesture recognition of daily life routines using 1D CNN," Frontiers in Bioengineering and Biotechnology, vol. 12, July 2024, Art. no. 1401803.

[24] Z. Gao, A. Sharma, M. Zheng, B. Planche, T. Chen, and Z. Wu, "Automated Patient Positioning with Learned 3D Hand Gestures," in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Feb. 2025, pp. 3772–3781.

[25] A. Adin, E. T. Krainski, A. Lenzi, Z. Liu, J. Martínez-Minaya, and H. Rue, "Automatic cross-validation in structured models: Is it time to leave out leave-one-out?," Spatial Statistics, vol. 62, Aug. 2024, Art. no. 100843.

[26] D. Gholamiangonabadi, N. Kiselov, and K. Grolinger, "Deep Neural Networks for Human Activity Recognition With Wearable Sensors: Leave-One-Subject-Out Cross-Validation for Model Selection," IEEE Access, vol. 8, pp. 133982–133994, 2020.

[27] S. Fathonah, E. Setyaningsih, E. Susanti, and Taukhit, "Cross-Patient Evaluation of CNN-Based Facial Expression Recognition for Intubated ICU Patients Using Leave-One-Patient-Out Validation," Engineering, Technology & Applied Science Research, vol. 16, no. 2, pp. 33187–33195, Apr. 2026.

[28] E. Setyaningsih, S. Fathonah, E. Susanti, and Tauhkit "Patient-independent Evaluation of CNN-LSTM for Facial Expression Recognition in Intubated ICU Patients," International Journal of Intelligent Engineering and Systems, vol. 19, no. 5, pp. 1109–1123, 2026.