Enhanced-PCA based Dimensionality Reduction and Feature Selection for Real-Time Network Threat Detection


  • P. More Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, India
  • P. Mishra Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, India
Volume: 10 | Issue: 5 | Pages: 6270-6275 | October 2020 | https://doi.org/10.48084/etasr.3801


With the rise of the data amount being collected and exchanged over networks, the threat of cyber-attacks has also increased significantly. Timely and accurate detection of any intrusion activity in networks has become a crucial task in order to safeguard data and other valuable assets. While manual moderation and programmed logic have been used for this purpose, the use of machine learning algorithms for superior pattern mapping is desired. The system logs in a network tend to include many parameters, and not all of them provide indications of an impending network threat. The selection of the right features is thus important for achieving better results. There is a need for accurate mapping of high dimension features to low dimension intermediate representations while retaining crucial information. In this paper, an approach for feature reduction and selection when working on the task of network threat detection is proposed. This approach modifies the traditional Principal Component Analysis (PCA) algorithm by working on its shortcomings and by minimizing the false detection rates. Specifically, work has been done upon the calculation of symmetric uncertainty and subsequent sorting of features. The performance of the proposed approach is evaluated on four standard-sized datasets that are collected using the Microsoft SYSMON real-time log collection tool. The proposed method is found to be better than the standard PCA and FAST methods for data reduction. The proposed approach makes a strong case as a dimensionality reduction and feature selection technique for minimizing false detection rates when operating on real-time data.


principal component analysis, fast clustering, dimensionality reduction, machine learning, network security


Download data is not yet available.


S. Staniford-Chen and L. T. Heberlein, "Holding intruders accountable on the Internet," in IEEE Symposium on Security and Privacy, Oakland, CA, USA, May 1995, pp. 39-49.

S.-J. Horng et al., "A novel intrusion detection system based on hierarchical clustering and support vector machines," Expert Systems with Applications, vol. 38, no. 1, pp. 306-313, Jan. 2011. DOI: https://doi.org/10.1016/j.eswa.2010.06.066

M. L. Shyu, S. C. Chen, K. Sarinnapakorn, and L. . W. Chang, "A Novel Anomaly Detection Scheme Based on Principal Component Classifier," 2003, pp. 172-179.

H. Ringberg, A. Soule, J. Rexford, and C. Diot, "Sensitivity of PCA for traffic anomaly detection," in ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, New York, NY, USA, Jun. 2007, pp. 109-120. DOI: https://doi.org/10.1145/1269899.1254895

V. Chandola, A. Banerjee, and V. Kumar, "Anomaly detection: A survey," ACM Computing Surveys, vol. 41, no. 3, pp. 1-58, Jul. 2009. DOI: https://doi.org/10.1145/1541880.1541882

H.-P. Kriegel, M. Schubert, and A. Zimek, "Angle-based outlier detection in high-dimensional data," in 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, Aug. 2008, pp. 444-452. DOI: https://doi.org/10.1145/1401890.1401946

X. Song, M. Wu, C. Jermaine, and S. Ranka, "Conditional Anomaly Detection," IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 5, pp. 631-645, May 2007. DOI: https://doi.org/10.1109/TKDE.2007.1009

M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, "LOF: identifying density-based local outliers," in ACM SIGMOD International Conference on Management of Data, New York, NY, USA, May 2000, pp. 93-104. DOI: https://doi.org/10.1145/335191.335388

A. T. Siahmarzkooh, S. Tabarsa, Z. H. Nasab, and F. Sedighi, "An Optimized Genetic Algorithm with Classification Approach used for Intrusion Detection," 2015. /paper/An-Optimized-Genetic-Algorithm-with-Classification-Siahmarzkooh-Tabarsa/b0e239298e7c6d8aa0e813a12fe55a2d12673e29 (accessed Sep. 12, 2020).

W. Dumouchel and M. Schonlau, "A Comparison of Test Statistics for Computer Intrusion Detection Based on Principal Components Regression of Transition Probabilities," in Proceedings of the 30th Symposium on the Interface: Computing Science and Statistics, 1998, pp. 404-413.

Z. Muda, W. Yassin, M. N. Sulaiman, and N. I. Udzir, "Intrusion detection based on K-Means clustering and Naïve Bayes classification," in 7th International Conference on Information Technology in Asia, Kuching, Sarawak, Malaysia, Jul. 2011, pp. 1-6. DOI: https://doi.org/10.1109/CITA.2011.5999520

A. T. Siahmarzkooh, J. Karimpour, and S. Lotfi, "A Cluster-based Approach Towards Detecting and Modeling Network Dictionary Attacks," Engineering, Technology & Applied Science Research, vol. 6, no. 6, pp. 1227-1234, Dec. 2016. DOI: https://doi.org/10.48084/etasr.937

J. Karimpour, S. Lotfi, and A. T. Siahmarzkooh, "Intrusion detection in network flows based on an optimized clustering criterion," Turkish Journal of Electrical Engineering & Computer Sciences, vol. 25, no. 3, pp. 1963-1975, May 2017. DOI: https://doi.org/10.3906/elk-1601-105

A. T. Siahmarzkooh, In press. A GWO-based Attack Detection System Using K-means Clustering Algorithm (No. TRKU-11-08-2020-10987), Technology Reports of Kansai University.

A. Lakhina, M. Crovella, and C. Diot, "Characterization of network-wide anomalies in traffic flows," in 4th ACM SIGCOMM Conference on Internet Measurement, New York, NY, USA, Oct. 2004, pp. 201-206. DOI: https://doi.org/10.1145/1028788.1028813

C. Taylor and J. Alves-Foss, "NATE: Network Analysis of Anomalous Traffic Events, a low-cost approach," in Proceedings of the 2001 workshop on New security paradigms, New York, NY, USA, Sep. 2001, pp. 89-96. DOI: https://doi.org/10.1145/508171.508186

C. Taylor and J. Alves-Foss, "An empirical analysis of NATE: Network Analysis of Anomalous Traffic Events," in Proceedings of the 2002 workshop on New security paradigms, New York, NY, USA, Sep. 2002, pp. 18-26. DOI: https://doi.org/10.1145/844102.844106

W. Wang and R. Battiti, "Identifying intrusions in computer networks with principal component analysis," in First International Conference on Availability, Reliability and Security, Vienna, Austria, Apr. 2006, pp. 1-8. DOI: https://doi.org/10.1109/ARES.2006.73

C. Callegari, L. Gazzarrini, S. Giordano, M. Pagano, and T. Pepe, "When randomness improves the anomaly detection performance," in 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies, Rome, Italy, Nov. 2010, pp. 1-5. DOI: https://doi.org/10.1109/ISABEL.2010.5702782

R. Kwitt and U. Hofmann, "Unsupervised Anomaly Detection in Network Traffic by Means of Robust PCA," in International Multi-Conference on Computing in the Global Information Technology, Guadeloupe City, Guadeloupe, Mar. 2007, pp. 37-37. DOI: https://doi.org/10.1109/ICCGI.2007.62

W. Lee and S. J. Stolfo, "A framework for constructing features and models for intrusion detection systems," ACM Transactions on Information and System Security, vol. 3, no. 4, pp. 227-261, Nov. 2000. DOI: https://doi.org/10.1145/382912.382914

M. Koeman, J. Engel, J. Jansen, and L. Buydens, "Critical comparison of methods for fault diagnosis in metabolomics data," Scientific Reports, vol. 9, no. 1, Feb. 2019, Art. no. 1123. DOI: https://doi.org/10.1038/s41598-018-37494-7

H. Zou, T. Hastie, and R. Tibshirani, "Sparse Principal Component Analysis," Journal of Computational and Graphical Statistics, vol. 15, no. 2, pp. 265-286, Jun. 2006. DOI: https://doi.org/10.1198/106186006X113430

N. T. Pham, E. Foo, S. Suriadi, H. Jeffrey, and H. F. M. Lahza, "Improving performance of intrusion detection system using ensemble methods and feature selection," in Proceedings of the Australasian Computer Science Week Multiconference, New York, NY, USA, Jan. 2018, pp. 1-6. DOI: https://doi.org/10.1145/3167918.3167951

A. J. Malik, W. Shahzad, and F. A. Khan, "Network intrusion detection using hybrid binary PSO and random forests algorithm," Security and Communication Networks, vol. 8, no. 16, pp. 2646-2660, 2015. DOI: https://doi.org/10.1002/sec.508

Y. Zhong et al., "HELAD: A novel network anomaly detection model based on heterogeneous ensemble learning," Computer Networks, vol. 169, Mar. 2020, Art. no. 107049. DOI: https://doi.org/10.1016/j.comnet.2019.107049

F. Rezaei and A. Zahedi, "Dealing with Wormhole Attacks in Wireless Sensor Networks Through Discovering Separate Routes Between Nodes," Engineering, Technology & Applied Science Research, vol. 7, no. 4, pp. 1771-1774, Aug. 2017. DOI: https://doi.org/10.48084/etasr.1118

P. Ratadiya and R. Moorthy, "Spam filtering on forums: A synthetic oversampling based approach for imbalanced data classification," arXiv:1909.04826 [cs, stat], Sep. 2019, Accessed: Sep. 12, 2020. [Online]. Available: http://arxiv.org/abs/1909.04826.

P. More and M. P. Mishra, "Machine Learning for Cyber Threat Detection," International Journal of Advanced Trends in Computer Science and Engineering, vol. 9, no. 1.1, pp. 41-46, 2020. DOI: https://doi.org/10.30534/ijatcse/2020/0891.12020

M. Russinovich and T. Garnier, "Sysmon v11.11," Jul. 15, 2020. https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon (accessed Sep. 12, 2020).


How to Cite

P. More and P. Mishra, “Enhanced-PCA based Dimensionality Reduction and Feature Selection for Real-Time Network Threat Detection”, Eng. Technol. Appl. Sci. Res., vol. 10, no. 5, pp. 6270–6275, Oct. 2020.


Abstract Views: 756
PDF Downloads: 508

Metrics Information