A Multimodal Cluster-Aware Learning Framework for Pilot Cognitive-State Recognition Using EEG and Peripheral Physiological Signals

Quynh Anh Nguyen; Nam Anh Dao

doi:10.48084/etasr.18805

Authors

Quynh Anh Nguyen Faculty of Information Technology, Electric Power University, Hanoi, Vietnam
Nam Anh Dao Faculty of Information Technology, Electric Power University, Hanoi, Vietnam

Volume: 16 | Issue: 4 | Pages: 37306-37316 | August 2026 | https://doi.org/10.48084/etasr.18805

Received: 17 March 2026 | Revised: 19 April 2026 and 1 May 2026 | Accepted: 3 May 2026 | Online: 21 June 2026

Corresponding author: Quynh Anh Nguyen

Abstract

Loss of state awareness remains a major contributor to aviation incidents, motivating the need for reliable and interpretable monitoring of pilot cognitive states. This study presents a multimodal cluster-aware learning framework for pilot cognitive-state recognition using Electroencephalography (EEG), Electrocardiography (ECG), Galvanic Skin Response (GSR), and respiration signals. The framework combines modality-aware preprocessing, hybrid dimensionality reduction through principal component analysis and an autoencoder, latent-structure discovery via k-means clustering, and cluster-specific classification using Light Gradient Boosting Machine (LGBM). Experiments were conducted on the public Kaggle benchmark Reducing Commercial Aviation Fatalities, which contains synchronized psychophysiological recordings from 18 pilots under four induced mental states. Under a stratified 5-fold benchmark protocol, the proposed method achieved an overall accuracy of 0.956, outperforming XGBoost, multilayer perceptron, and several representative baselines. The results indicate that multimodal fusion and cluster-aware classification improve the representation of heterogeneous psychophysiological patterns while maintaining computational tractability. Although additional subject-independent validation remains necessary to establish stronger generalization claims, the proposed framework provides a structured and scalable foundation for cognitive-state monitoring in aviation environments.

Keywords:

cognitive state detection, multimodal physiological signals, EEG-based monitoring, aviation safety, machine learning, dimensionality reduction, clustering methods

References

[1] F. L. Da Silva, "EEG: Origin and Measurement," in EEG - fMRI, C. Mulert and L. Lemieux, Eds. Springer Berlin Heidelberg, 2009, pp. 19–38.

[2] M. Martinussen and D. R. Hunter, Aviation Psychology and Human Factors, 2nd ed. Taylor & Francis, CRC Press, 2017.

[3] D. Kelly and M. Efthymiou, "An analysis of human factors in fifty controlled flight into terrain aviation accidents from 2007 to 2017," Journal of Safety Research, vol. 69, pp. 155–165, June 2019.

[4] B. A. Hamilton, "Reducing Commercial Aviation Fatalities." https://kaggle.com/reducing-commercial-aviation-fatalities.

[5] E. Van Weelden, M. Alimardani, T. J. Wiltshire, and M. M. Louwerse, "Aviation and neurophysiology: A systematic review," Applied Ergonomics, vol. 105, Nov. 2022, Art. no. 103838.

[6] I. Alreshidi, I. Moulitsas, and K. W. Jenkins, "Multimodal Approach for Pilot Mental State Detection Based on EEG," Sensors, vol. 23, no. 17, Aug. 2023, Art. no. 7350.

[7] I. Alreshidi, D. Bisandu, and I. Moulitsas, "Illuminating the Neural Landscape of Pilot Mental States: A Convolutional Neural Network Approach with Shapley Additive Explanations Interpretability," Sensors, vol. 23, no. 22, Nov. 2023, Art. no. 9052.

[8] A. R. Harrivel, C. Liles, C. L. Stephens, K. K. Ellis, L. J. Prinzel, and A. T. Pope, "Psychophysiological Sensing and State Classification for Attention Management in Commercial Aviation," in AIAA Infotech @ Aerospace, Jan. 2016.

[9] E. Q. Wu et al., "Detecting Fatigue Status of Pilots Based on Deep Learning Network Using EEG Signals," IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 3, pp. 575–585, Sept. 2021.

[10] D. H. Lee, J. H. Jeong, B. W. Yu, T. E. Kam, and S. W. Lee, "Autonomous System for EEG-Based Multiple Abnormal Mental States Classification Using Hybrid Deep Neural Networks Under Flight Environment," IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 10, pp. 6426–6437, Oct. 2023.

[11] X. Wang, G. Gong, N. Li, L. Ding, and Y. Ma, "Decoding pilot behavior consciousness of EEG, ECG, eye movements via an SVM machine learning model," International Journal of Modeling, Simulation, and Scientific Computing, vol. 11, no. 04, Aug. 2020, Art. no. 2050028.

[12] Q. Wang, Z. Wang, R. Xiong, X. Liao, and X. Tan, "A Method for Classification and Evaluation of Pilot’s Mental States Based on CNN," Computer Systems Science and Engineering, vol. 46, no. 2, pp. 1999–2020, 2023.

[13] M. J. Hasan, D. Shon, K. Im, H.-K. Choi, D.-S. Yoo, and J.-M. Kim, "Sleep State Classification Using Power Spectral Density and Residual Neural Network with Multichannel EEG Signals," Applied Sciences, vol. 10, no. 21, Oct. 2020, Art. no. 7639.

[14] I. T. Jolliffe and J. Cadima, "Principal component analysis: a review and recent developments," Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 374, no. 2065, Apr. 2016, Art. no. 20150202.

[15] S. R. Jeevakala and H. Ramasangu, "Classification of Cognitive States using Task-Specific Connectivity Features," Engineering, Technology & Applied Science Research, vol. 13, no. 3, pp. 10675–10679, June 2023.

[16] Y. Jia et al., "Preprocessing and Denoising Techniques for Electrocardiography and Magnetocardiography: A Review," Bioengineering, vol. 11, no. 11, Nov. 2024, Art. no. 1109.

[17] K. Berahmand, F. Daneshfar, E. S. Salehi, Y. Li, and Y. Xu, "Autoencoders and their applications in machine learning: a survey," Artificial Intelligence Review, vol. 57, no. 2, Feb. 2024, Art. no. 28.

[18] S. Lloyd, "Least squares quantization in PCM," IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 129–137, Mar. 1982.

[19] G. Ke et al., "LightGBM: A Highly Efficient Gradient Boosting Decision Tree," in Advances in Neural Information Processing Systems, 2017, vol. 30.