Enhancing Network Intrusion Detection for TLS Traffic Using Deep Learning
Received: 9 July 2025 | Revised: 13 August 2025 and 25 August 2025 | Accepted: 30 August 2025 | Online: 8 December 2025
Corresponding author: Hidayatul Muttaqien
Abstract
The increased utilization of Transport Layer Security (TLS) encryption in contemporary network communication introduces new obstacles for Network Intrusion Detection Systems (NIDS), since encrypted traffic constrains the efficacy of traditional signature-based techniques. This study presents a real-time intrusion detection method for TLS traffic utilizing a combination of Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTM) networks. CNNs are employed to derive spatial representations of TLS information from Suricata logs, including JA3 fingerprints, cipher suites, and connection statistics, and BiLSTM is utilized to capture bidirectional temporal dependencies of encrypted traffic to identify intricate anomaly patterns. This model was evaluated utilizing an extensive TLS dataset comprising both valid and malicious traffic, including Command-and-Control (C2) connections, malware communication, and data exfiltration. The experimental findings indicate that the CNN–BiLSTM model attained a detection accuracy of 98.7%, a False Positive Rate (FPR) of 1.4%, and an average processing time of 12.9 ms per session, rendering it appropriate for real-time application in corporate network security systems. This methodology enhances the capability of hybrid Deep Learning (DL) models to identify concealed dangers in TLS communication without requiring data decryption.
Keywords:
Transport Layer Security (TLS), Network Intrusion Detection Systems (NIDS), Deep Learning (DL), Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), encrypted traffic, SuricataDownloads
References
R. Mohammad, F. Saeed, A. A. Almazroi, F. S. Alsubaei, and A. A. Almazroi, "Enhancing Intrusion Detection Systems Using a Deep Learning and Data Augmentation Approach," Systems, vol. 12, no. 3, Mar. 2024, Art. no. 79. DOI: https://doi.org/10.3390/systems12030079
A. A. Ghani and S. A. Alasadi, "A Deep Learning Algorithm to Cybersecurity: Enhancing Intrusion Detection with a Hybrid GRU and BiLSTM Model," Engineering, Technology & Applied Science Research, vol. 15, no. 3, pp. 23605–23612, Jun. 2025. DOI: https://doi.org/10.48084/etasr.10666
B. Anderson, S. Paul, and D. McGrew, "Deciphering malware’s use of TLS (without decryption)," Journal of Computer Virology and Hacking Techniques, vol. 14, no. 3, pp. 195–211, Aug. 2018. DOI: https://doi.org/10.1007/s11416-017-0306-6
R. H. Altaie and H. K. Hoomod, "An Intrusion Detection System using a Hybrid Lightweight Deep Learning Algorithm," Engineering, Technology & Applied Science Research, vol. 14, no. 5, pp. 16740–16743, Oct. 2024. DOI: https://doi.org/10.48084/etasr.7657
M. Jouhari and M. Guizani, "Lightweight CNN-BiLSTM based Intrusion Detection Systems for Resource-Constrained IoT Devices," in 2024 International Wireless Communications and Mobile Computing, Ayia Napa, Cyprus, 2024, pp. 1558–1563. DOI: https://doi.org/10.1109/IWCMC61514.2024.10592352
J. Zhang, X. Zhang, Z. Liu, F. Fu, Y. Jiao, and F. Xu, "A Network Intrusion Detection Model Based on BiLSTM with Multi-Head Attention Mechanism," Electronics, vol. 12, no. 19, Oct. 2023, Art. no. 4170. DOI: https://doi.org/10.3390/electronics12194170
H. Benaddi, M. Jouhari, and O. Elharrouss, "A lightweight hybrid approach for intrusion detection systems using a chi-square feature selection approach in IoT," Internet of Things, vol. 32, Jul. 2025, Art. no. 101624. DOI: https://doi.org/10.1016/j.iot.2025.101624
X. Qiu, G. Yan, and L. Yin, "CLSTM-MT (a Combination of 2-Conv CNN and BiLSTM Under the Mean Teacher Collaborative Learning Framework): Encryption Traffic Classification Based on CLSTM (a Combination of 2-Conv CNN and BiLSTM) and Mean Teacher Collaborative Learning," Applied Sciences, vol. 15, no. 9, May 2025, Art. no. 5089. DOI: https://doi.org/10.3390/app15095089
Y. Imrana, Y. Xiang, L. Ali, and Z. Abdul-Rauf, "A bidirectional LSTM deep learning approach for intrusion detection," Expert Systems with Applications, vol. 185, Dec. 2021, Art. no. 115524. DOI: https://doi.org/10.1016/j.eswa.2021.115524
R. Ben Said, Z. Sabir, and I. Askerzade, "CNN-BiLSTM: A Hybrid Deep Learning Approach for Network Intrusion Detection System in Software-Defined Networking With Hybrid Feature Selection," IEEE Access, vol. 11, pp. 138732–138747, 2023. DOI: https://doi.org/10.1109/ACCESS.2023.3340142
A. N. Abdullah, "Development of an Intrusion Detection System using an Ensemble Voting Machine Learning Technique," Engineering, Technology & Applied Science Research, vol. 15, no. 3, pp. 23917–23922, Jun. 2025. DOI: https://doi.org/10.48084/etasr.10764
F. Amiri, M. Rezaei Yousefi, C. Lucas, A. Shakery, and N. Yazdani, "Mutual information-based feature selection for intrusion detection systems," Journal of Network and Computer Applications, vol. 34, no. 4, pp. 1184–1199, Jul. 2011. DOI: https://doi.org/10.1016/j.jnca.2011.01.002
M. S. Akhtar and T. Feng, "Deep Learning-Based Framework for the Detection of Cyberattack Using Feature Engineering," Security and Communication Networks, vol. 2021, no. 1, Dec. 2021, Art. no. 6129210. DOI: https://doi.org/10.1155/2021/6129210
K. Mala and H. S. Annapurna, "The Zoneout Regularized Gated Recurrent Unit Algorithm for Network Intrusion Detection with Class Imbalance Mitigation," Engineering, Technology & Applied Science Research, vol. 15, no. 4, pp. 24758–24763, Aug. 2025. DOI: https://doi.org/10.48084/etasr.11823
A.-A. Maiga, E. Ataro, and S. Githinji, "Balancing Data Privacy and 5G VNFs Security Monitoring: Federated Learning with CNN + BiLSTM + LSTM Model," Journal of Electrical and Computer Engineering, vol. 2024, no. 1, Mar. 2024, Art. no. 5134326. DOI: https://doi.org/10.1155/2024/5134326
N. O. Aljehane, H. A. Mengash, S. B. H. Hassine, F. A. Alotaibi, A. S. Salama, and S. Abdelbagi, "Optimizing intrusion detection using intelligent feature selection with machine learning model," Alexandria Engineering Journal, vol. 91, pp. 39–49, Mar. 2024. DOI: https://doi.org/10.1016/j.aej.2024.01.073
S. Z. Majidian, S. TaghipourEivazi, B. Arasteh, and A. Ghaffari, "Optimizing random forests to detect intrusion in the Internet of Things," Computers and Electrical Engineering, vol. 120, Dec. 2024, Art. no. 109860. DOI: https://doi.org/10.1016/j.compeleceng.2024.109860
H. Liu, M. Zhou, and Q. Liu, "An embedded feature selection method for imbalanced data classification," IEEE/CAA Journal of Automatica Sinica, vol. 6, no. 3, pp. 703–715, May 2019. DOI: https://doi.org/10.1109/JAS.2019.1911447
K. Ren, Y. Zeng, Z. Cao, and Y. Zhang, "ID-RDRL: a deep reinforcement learning-based feature selection intrusion detection model," Scientific Reports, vol. 12, no. 1, Sep. 2022, Art. no. 15370. DOI: https://doi.org/10.1038/s41598-022-19366-3
J. Lansky et al., "Deep Learning-Based Intrusion Detection Systems: A Systematic Review," IEEE Access, vol. 9, pp. 101574–101599, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3097247
S. Elsayed, K. Mohamed, and M. A. Madkour, "A Comparative Study of Using Deep Learning Algorithms in Network Intrusion Detection," IEEE Access, vol. 12, pp. 58851–58870, 2024. DOI: https://doi.org/10.1109/ACCESS.2024.3389096
T.-P. Nguyen, J. Cho, and D. Kim, "Semi-supervised intrusion detection system for in-vehicle networks based on variational autoencoder and adversarial reinforcement learning," Knowledge-Based Systems, vol. 304, Nov. 2024, Art. no. 112563. DOI: https://doi.org/10.1016/j.knosys.2024.112563
P. Soltanzadeh and M. Hashemzadeh, "RCSMOTE: Range-Controlled synthetic minority over-sampling technique for handling the class imbalance problem," Information Sciences, vol. 542, pp. 92–111, Jan. 2021. DOI: https://doi.org/10.1016/j.ins.2020.07.014
X. Lin, G. Xiong, G. Gou, Z. Li, J. Shi, and J. Yu, "ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification," in Proceedings of the ACM Web Conference 2022, Virtual Event, Lyon, France, 2022, pp. 633–642. DOI: https://doi.org/10.1145/3485447.3512217
P. R. Kanna and P. Santhi, "Hybrid Intrusion Detection using MapReduce based Black Widow Optimized Convolutional Long Short-Term Memory Neural Networks," Expert Systems with Applications, vol. 194, May 2022, Art. no. 116545. DOI: https://doi.org/10.1016/j.eswa.2022.116545
Kirubavathi. G and A. R. Nair, "Hybrid Deep Learning framework-based intrusion detection system for the Internet of Things," in 2024 International Conference on Intelligent Systems for Cybersecurity, Gurugram, India, 2024, pp. 1–6. DOI: https://doi.org/10.1109/ISCS61804.2024.10581228
R. Vinayakumar, K. P. Soman, and P. Poornachandran, "Applying convolutional neural network for network intrusion detection," in 2017 International Conference on Advances in Computing, Communications and Informatics, Udupi, India, 2017, pp. 1222–1228. DOI: https://doi.org/10.1109/ICACCI.2017.8126009
S. M. Kasongo, "A deep learning technique for intrusion detection system using a Recurrent Neural Networks based framework," Computer Communications, vol. 199, pp. 113–125, Feb. 2023. DOI: https://doi.org/10.1016/j.comcom.2022.12.010
A. Bhardwaj, R. Tyagi, N. Sharma, A. Khare, M. S. Punia, and V. K. Garg, "Network intrusion detection in software defined networking with self-organized constraint-based intelligent learning framework," Measurement: Sensors, vol. 24, Dec. 2022, Art. no. 100580. DOI: https://doi.org/10.1016/j.measen.2022.100580
M. Rahouti, K. Xiong, Y. Xin, S. K. Jagatheesaperumal, M. Ayyash, and M. Shaheed, "SDN Security Review: Threat Taxonomy, Implications, and Open Challenges," IEEE Access, vol. 10, pp. 45820–45854, 2022. DOI: https://doi.org/10.1109/ACCESS.2022.3168972
I. Aliyu, M. C. Feliciano, S. Van Engelenburg, D. O. Kim, and C. G. Lim, "A Blockchain-Based Federated Forest for SDN-Enabled In-Vehicle Network Intrusion Detection System," IEEE Access, vol. 9, pp. 102593–102608, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3094365
A. Chiras, A. Peratikou, and S. Stavrou, "Increasing Security of Containerized Blockchain using SDN," in 2024 Panhellenic Conference on Electronics & Telecommunications, Thessaloniki, Greece, 2024, pp. 1–5. DOI: https://doi.org/10.1109/PACET60398.2024.10497057
S. Faizullah, M. A. Khan, A. Alzahrani, and I. Khan, "Permissioned Blockchain-Based Security for SDN in IoT Cloud Networks," in 2019 International Conference on Advances in the Emerging Computing Technologies, Al Madinah Al Munawwarah, Saudi Arabia, 2020, pp. 1–6. DOI: https://doi.org/10.1109/AECT47998.2020.9194181
M. H. Rifat, A. Islam Ananna, T. Intesir Ahmed, S. Akter, and N. Mansoor, "Blockchain-Based Controller Recovery and SDN Packet Filtering Scheme for Softwarized UAVs," in 2024 International Conference on Advances in Computing, Communication, Electrical, and Smart Systems, Dhaka, Bangladesh, 2024, pp. 1–5. DOI: https://doi.org/10.1109/iCACCESS61735.2024.10499453
C. Kaushik, D. VarunTeja, M. S. Krishna, and S. Jaavali, "DDoS Attack Detection and Mitigation Using Mininet and RYU Controller in SDN Environment," in 2024 15th International Conference on Computing Communication and Networking Technologies, Kamand, India, 2024, pp. 1–7. DOI: https://doi.org/10.1109/ICCCNT61001.2024.10724700
A. Jain, D. Kumar Khatri, A. Ayyagiri, C. Mokkapati, V. B. R. Bhimanapati, and L. H. Alzubaidi, "Secure and Scalable IoT Networks: Optimizing Blockchain and SDN for Smart Environments," in 2024 4th International Conference on Blockchain Technology and Information Security, Wuhan, China, 2024, pp. 338–344. DOI: https://doi.org/10.1109/ICBCTIS64495.2024.00060
F. Pacheco, E. Exposito, M. Gineste, C. Baudoin, and J. Aguilar, "Towards the Deployment of Machine Learning Solutions in Network Traffic Classification: A Systematic Survey," IEEE Communications Surveys & Tutorials, vol. 21, no. 2, pp. 1988–2014, Apr. 2019. DOI: https://doi.org/10.1109/COMST.2018.2883147
M. Shen et al., "Machine Learning-Powered Encrypted Network Traffic Analysis: A Comprehensive Survey," IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 791–824, Jan. 2023. DOI: https://doi.org/10.1109/COMST.2022.3208196
B. Yan et al., "Graph Mining for Cybersecurity: A Survey," ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 2, Nov. 2023, Art. no. 47. DOI: https://doi.org/10.1145/3610228
S. Rezaei and X. Liu, "Deep Learning for Encrypted Traffic Classification: An Overview," IEEE Communications Magazine, vol. 57, no. 5, pp. 76–81, May 2019. DOI: https://doi.org/10.1109/MCOM.2019.1800819
B. Ji et al., "Survey of Secure Communications of Internet of Things with Artificial Intelligence," IEEE Internet of Things Magazine, vol. 5, no. 3, pp. 92–99, Sep. 2022. DOI: https://doi.org/10.1109/IOTM.001.2100178
Ö. Aslan, S. S. Aktuğ, M. Ozkan-Okay, A. A. Yilmaz, and E. Akin, "A Comprehensive Review of Cyber Security Vulnerabilities, Threats, Attacks, and Solutions," Electronics, vol. 12, no. 6, Mar. 2023, Art. no. 1333. DOI: https://doi.org/10.3390/electronics12061333
A. Thakkar and R. Lohiya, "A survey on intrusion detection system: feature selection, model, performance measures, application perspective, challenges, and future research directions," Artificial Intelligence Review, vol. 55, no. 1, pp. 453–563, Jan. 2022. DOI: https://doi.org/10.1007/s10462-021-10037-9
M. S. K. K, T. Sree, S. V. D, Y. S. S. Harsha, and N. Rajagopalan, "Suricata-Based Intrusion Detection and Isolation System for Local Area Networks," in 2024 International Conference on Signal Processing, Computation, Electronics, Power and Telecommunication, Karaikal, India, 2024, pp. 1–5. DOI: https://doi.org/10.1109/IConSCEPT61884.2024.10627890
B. Omarov, O. Auelbekov, A. Suliman, and A. Zhaxanova, "CNN-BiLSTM Hybrid Model for Network Anomaly Detection in Internet of Things," International Journal of Advanced Computer Science and Applications, vol. 14, no. 3, pp. 436–444, Mar. 2023. DOI: https://doi.org/10.14569/IJACSA.2023.0140349
Z. Liu et al., "Using Embedded Feature Selection and CNN for Classification on CCD-INID-V1—A New IoT Dataset," Sensors, vol. 21, no. 14, Jul. 2021, Art. no. 4834. DOI: https://doi.org/10.3390/s21144834
M. S. Alshehri, J. Ahmad, S. Almakdi, M. A. Qathrady, Y. Y. Ghadi, and W. J. Buchanan, "SkipGateNet: A Lightweight CNN-LSTM Hybrid Model With Learnable Skip Connections for Efficient Botnet Attack Detection in IoT," IEEE Access, vol. 12, pp. 35521–35538, 2024. DOI: https://doi.org/10.1109/ACCESS.2024.3371992
S. Ebadinezhad, N. N. Nia, N. Shirzad, and N. K. Osemeha, "Enhancing Intrusion Detection Systems Using RNN, LSTM, and Hybrid RNN-LSTM Models," in 2025 International Conference on Machine Learning and Autonomous Systems, Prawet, Thailand, 2025, pp. 1108–1115. DOI: https://doi.org/10.1109/ICMLAS64557.2025.10968214
A. A. A. Mohammed, "Improving Intrusion Detection Systems by Using Deep Learning Methods on Time Series Data," Engineering, Technology & Applied Science Research, vol. 15, no. 1, pp. 19267–19272, Feb. 2025. DOI: https://doi.org/10.48084/etasr.9417
H. Y. I. Khalid and N. B. I. Aldabagh, "A Survey on the Latest Intrusion Detection Datasets for Software Defined Networking Environments," Engineering, Technology & Applied Science Research, vol. 14, no. 2, pp. 13190–13200, Apr. 2024. DOI: https://doi.org/10.48084/etasr.6756
M. H. Bhavsar, Y. B. Bekele, K. Roy, J. C. Kelly, and D. Limbrick, "FL-IDS: Federated Learning-Based Intrusion Detection System Using Edge Devices for Transportation IoT," IEEE Access, vol. 12, pp. 52215–52226, 2024. DOI: https://doi.org/10.1109/ACCESS.2024.3386631
H. Muttaqien, "Suricata Logs." Kaggle. [Online]. Available: www.kaggle.com/datasets/muttaqien19/dataset-suricata-logs.
A. Ferriyan, A. H. Thamrin, K. Takeda, and J. Murai, "Generating Network Intrusion Detection Dataset Based on Real and Encrypted Synthetic Attack Traffic," Applied Sciences, vol. 11, no. 17, Sep. 2021, Art. no. 7868. DOI: https://doi.org/10.3390/app11177868
S. N. and A. Haldorai, "Efficient Intrusion Detection System Data Preprocessing Using Deep Sparse Autoencoder with Differential Evolution," IET Information Security, vol. 2024, no. 1, Aug. 2024, Art. no. 9937803. DOI: https://doi.org/10.1049/2024/9937803
Downloads
How to Cite
License
Copyright (c) 2025 Hidayatul Muttaqien, Muhammad Niswar, Syafruddin Syarif, Zahir Zainuddin
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
