A Reliability-Based Multimodal Framework for 3D Face Recognition under Occlusion

M. L. Gangadhar; A. S. Raju; C. R. Roopashree

doi:10.48084/etasr.19247

Authors

M. L. Gangadhar Department of Information Science and Engineering, Sri Siddhartha Institute of Technology, Sri Siddhartha Academy of Higher Education, Tumakuru, Karnataka, India
A. S. Raju Department of Biomedical Engineering, Sri Siddhartha Institute of Technology, Sri Siddhartha Academy of Higher Education, Tumakuru, Karnataka, India
C. R. Roopashree Department of Mechanical Engineering, Sri Siddhartha Institute of Technology, Sri Siddhartha Academy of Higher Education, Tumakuru, Karnataka, India

Volume: 16 | Issue: 4 | Pages: 37869-37874 | August 2026 | https://doi.org/10.48084/etasr.19247

Received: 10 April 2026 | Revised: 5 May 2026, 24 May 2026, 25 May 2026, 27 May 2026, and 28 May 2026 | Accepted: 30 May 2026 | Online: 20 June 2026

Corresponding author: M. L. Gangadhar

Abstract

Face recognition in real-world and uncontrolled environments is greatly affected when the face is partially covered by masks, glasses, scarves, hair, or due to pose-related self-occlusion. Although three-dimensional (3D) face recognition is generally more robust to lighting changes and moderate pose variations, its performance still reduces when important facial regions are heavily covered. To overcome this problem, this study presents an occlusion-aware hybrid biometric framework for reliable 3D face recognition. The proposed method combines reconstructed 3D shape features, deep texture features, and additional biometric cues using an adaptive weighted fusion approach. An occlusion detection and generative reconstruction module is used to recover missing facial regions before feature extraction, and an attention mechanism reduces the impact of unreliable areas while focusing on important facial features. Extensive experiments on benchmark datasets, such as Bosphorus, BU-3DFE, and FRGC v2.0, show that the proposed framework performs better than strong single-modality and traditional multimodal methods. The proposed system reaches an accuracy of up to 98.7%, even in partial occlusions, and significantly reduces the Equal Error Rate (EER), demonstrating its effectiveness and suitability for real-world biometric authentication applications.

Keywords:

partial occlusion, deep learning, 3D face recognition, hybrid biometrics, multimodal fusion

References

[1] X. Tan, S. Chen, Z. H. Zhou, and F. Zhang, "Face recognition from a single image per person: A survey," Pattern Recognition, vol. 39, no. 9, pp. 1725–1745, Sept. 2006.

[2] J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma, "Robust Face Recognition via Sparse Representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210–227, Oct. 2009.

[3] W. Zhe et al., "A Research on Two-Stage Facial Occlusion Recognition Algorithm based on CNN," Engineering, Technology & Applied Science Research, vol. 14, no. 6, pp. 18205–18212, Dec. 2024.

[4] A. Colombo, C. Cusano, and R. Schettini, "Three-Dimensional Occlusion Detection and Restoration of Partially Occluded Faces," Journal of Mathematical Imaging and Vision, vol. 40, no. 1, pp. 105–119, May 2011.

[5] X. Tan and B. Triggs, "Enhanced Local Texture Feature Sets for Face Recognition Under Difficult Lighting Conditions," IEEE Transactions on Image Processing, vol. 19, no. 6, pp. 1635–1650, June 2010.

[6] A. Ross and A. K. Jain, "Multimodal biometrics: An overview," in 2004 12th European Signal Processing Conference, Sept. 2004, pp. 1221–1224.

[7] M. De Marsico, M. Nappi, and D. Riccio, "Measuring sample distortions in face recognition," in Proceedings of the 2nd ACM workshop on Multimedia in forensics, security and intelligence, Oct. 2010, pp. 83–88.

[8] Y. Sun, X. Wang, and X. Tang, "Deep Learning Face Representation from Predicting 10,000 Classes," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, June 2014, pp. 1891–1898.

[9] I. Kemelmacher-Shlizerman and S. M. Seitz, "Face reconstruction in the wild," in 2011 International Conference on Computer Vision, Nov. 2011, pp. 1746–1753.

[10] S. Zhou and S. Xiao, "3D face recognition: a survey," Human-centric Computing and Information Sciences, vol. 8, no. 1, Dec. 2018, Art. no. 35.

[11] A. Savran et al., "Bosphorus Database for 3D Face Analysis," in Biometrics and Identity Management, vol. 5372, B. Schouten, N. C. Juul, A. Drygajlo, and M. Tistarelli, Eds. Springer Berlin Heidelberg, 2008, pp. 47–56.

[12] Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and M. J. Rosato, "A 3D Facial Expression Database For Facial Behavior Research," in 7th International Conference on Automatic Face and Gesture Recognition (FGR06), 2006, pp. 211–216.

[13] P. J. Phillips et al., "Overview of the Face Recognition Grand Challenge," in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, vol. 1, pp. 947–954.

[14] Y. M. Cheung, M. Li, and R. Zou, "Facial Structure Guided GAN for Identity-preserved Face Image De-occlusion," in Proceedings of the 2021 International Conference on Multimedia Retrieval, Aug. 2021, pp. 46–54.

[15] C. Wang et al., "Deep learning-based 3D reconstruction from multiple images: A survey," Neurocomputing, vol. 597, Sept. 2024, Art. no. 128018.

[16] J. Liu, W. Wang, J. Yu, C. Zhang, and Y. Su, "3DFP-FCGAN: Face completion generative adversarial network with 3D facial prior," Journal of Visual Communication and Image Representation, vol. 82, Jan. 2022, Art. no. 103380.

[17] C. Ding and D. Tao, "Robust Face Recognition via Multimodal Deep Face Representation," IEEE Transactions on Multimedia, vol. 17, no. 11, pp. 2049–2058, Nov. 2015.

[18] S. Z. Li and A. K. Jain, Eds., Handbook of Face Recognition. Springer, 2011.

[19] A. Vaswani et al., "Attention is All you Need," in Advances in Neural Information Processing Systems, 2017, vol. 30.

[20] F. Wang et al., "Residual Attention Network for Image Classification," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, pp. 6450–6458.

[21] S. Soleymani, A. Dabouei, F. Taherkhani, S. M. Iranmanesh, J. Dawson, and N. M. Nasrabadi, "Quality-Aware Multimodal Biometric Recognition," IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 4, no. 1, pp. 97–116, Jan. 2022.

[22] "3D Facial Expression Database," Binghamton University. https://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html.

[23] "Face Recognition Grand Challenge (FRGC)," NIST, [Online]. Available: https://www.nist.gov/programs-projects/face-recognition-grand-challenge-frgc.