Enhanced-PCA based Dimensionality Reduction and Feature Selection for Real-Time Network Threat Detection
Abstract
With the rise of the data amount being collected and exchanged over networks, the threat of cyber-attacks has also increased significantly. Timely and accurate detection of any intrusion activity in networks has become a crucial task in order to safeguard data and other valuable assets. While manual moderation and programmed logic have been used for this purpose, the use of machine learning algorithms for superior pattern mapping is desired. The system logs in a network tend to include many parameters, and not all of them provide indications of an impending network threat. The selection of the right features is thus important for achieving better results. There is a need for accurate mapping of high dimension features to low dimension intermediate representations while retaining crucial information. In this paper, an approach for feature reduction and selection when working on the task of network threat detection is proposed. This approach modifies the traditional Principal Component Analysis (PCA) algorithm by working on its shortcomings and by minimizing the false detection rates. Specifically, work has been done upon the calculation of symmetric uncertainty and subsequent sorting of features. The performance of the proposed approach is evaluated on four standard-sized datasets that are collected using the Microsoft SYSMON real-time log collection tool. The proposed method is found to be better than the standard PCA and FAST methods for data reduction. The proposed approach makes a strong case as a dimensionality reduction and feature selection technique for minimizing false detection rates when operating on real-time data.
Keywords:
principal component analysis, fast clustering, dimensionality reduction, machine learning, network securityDownloads
References
S. Staniford-Chen and L. T. Heberlein, "Holding intruders accountable on the Internet," in IEEE Symposium on Security and Privacy, Oakland, CA, USA, May 1995, pp. 39-49.
S.-J. Horng et al., "A novel intrusion detection system based on hierarchical clustering and support vector machines," Expert Systems with Applications, vol. 38, no. 1, pp. 306-313, Jan. 2011. DOI: https://doi.org/10.1016/j.eswa.2010.06.066
M. L. Shyu, S. C. Chen, K. Sarinnapakorn, and L. . W. Chang, "A Novel Anomaly Detection Scheme Based on Principal Component Classifier," 2003, pp. 172-179.
H. Ringberg, A. Soule, J. Rexford, and C. Diot, "Sensitivity of PCA for traffic anomaly detection," in ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, New York, NY, USA, Jun. 2007, pp. 109-120. DOI: https://doi.org/10.1145/1269899.1254895
V. Chandola, A. Banerjee, and V. Kumar, "Anomaly detection: A survey," ACM Computing Surveys, vol. 41, no. 3, pp. 1-58, Jul. 2009. DOI: https://doi.org/10.1145/1541880.1541882
H.-P. Kriegel, M. Schubert, and A. Zimek, "Angle-based outlier detection in high-dimensional data," in 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, Aug. 2008, pp. 444-452. DOI: https://doi.org/10.1145/1401890.1401946
X. Song, M. Wu, C. Jermaine, and S. Ranka, "Conditional Anomaly Detection," IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 5, pp. 631-645, May 2007. DOI: https://doi.org/10.1109/TKDE.2007.1009
M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander, "LOF: identifying density-based local outliers," in ACM SIGMOD International Conference on Management of Data, New York, NY, USA, May 2000, pp. 93-104. DOI: https://doi.org/10.1145/335191.335388
A. T. Siahmarzkooh, S. Tabarsa, Z. H. Nasab, and F. Sedighi, "An Optimized Genetic Algorithm with Classification Approach used for Intrusion Detection," 2015. /paper/An-Optimized-Genetic-Algorithm-with-Classification-Siahmarzkooh-Tabarsa/b0e239298e7c6d8aa0e813a12fe55a2d12673e29 (accessed Sep. 12, 2020).
W. Dumouchel and M. Schonlau, "A Comparison of Test Statistics for Computer Intrusion Detection Based on Principal Components Regression of Transition Probabilities," in Proceedings of the 30th Symposium on the Interface: Computing Science and Statistics, 1998, pp. 404-413.
Z. Muda, W. Yassin, M. N. Sulaiman, and N. I. Udzir, "Intrusion detection based on K-Means clustering and Naïve Bayes classification," in 7th International Conference on Information Technology in Asia, Kuching, Sarawak, Malaysia, Jul. 2011, pp. 1-6. DOI: https://doi.org/10.1109/CITA.2011.5999520
A. T. Siahmarzkooh, J. Karimpour, and S. Lotfi, "A Cluster-based Approach Towards Detecting and Modeling Network Dictionary Attacks," Engineering, Technology & Applied Science Research, vol. 6, no. 6, pp. 1227-1234, Dec. 2016. DOI: https://doi.org/10.48084/etasr.937
J. Karimpour, S. Lotfi, and A. T. Siahmarzkooh, "Intrusion detection in network flows based on an optimized clustering criterion," Turkish Journal of Electrical Engineering & Computer Sciences, vol. 25, no. 3, pp. 1963-1975, May 2017. DOI: https://doi.org/10.3906/elk-1601-105
A. T. Siahmarzkooh, In press. A GWO-based Attack Detection System Using K-means Clustering Algorithm (No. TRKU-11-08-2020-10987), Technology Reports of Kansai University.
A. Lakhina, M. Crovella, and C. Diot, "Characterization of network-wide anomalies in traffic flows," in 4th ACM SIGCOMM Conference on Internet Measurement, New York, NY, USA, Oct. 2004, pp. 201-206. DOI: https://doi.org/10.1145/1028788.1028813
C. Taylor and J. Alves-Foss, "NATE: Network Analysis of Anomalous Traffic Events, a low-cost approach," in Proceedings of the 2001 workshop on New security paradigms, New York, NY, USA, Sep. 2001, pp. 89-96. DOI: https://doi.org/10.1145/508171.508186
C. Taylor and J. Alves-Foss, "An empirical analysis of NATE: Network Analysis of Anomalous Traffic Events," in Proceedings of the 2002 workshop on New security paradigms, New York, NY, USA, Sep. 2002, pp. 18-26. DOI: https://doi.org/10.1145/844102.844106
W. Wang and R. Battiti, "Identifying intrusions in computer networks with principal component analysis," in First International Conference on Availability, Reliability and Security, Vienna, Austria, Apr. 2006, pp. 1-8. DOI: https://doi.org/10.1109/ARES.2006.73
C. Callegari, L. Gazzarrini, S. Giordano, M. Pagano, and T. Pepe, "When randomness improves the anomaly detection performance," in 3rd International Symposium on Applied Sciences in Biomedical and Communication Technologies, Rome, Italy, Nov. 2010, pp. 1-5. DOI: https://doi.org/10.1109/ISABEL.2010.5702782
R. Kwitt and U. Hofmann, "Unsupervised Anomaly Detection in Network Traffic by Means of Robust PCA," in International Multi-Conference on Computing in the Global Information Technology, Guadeloupe City, Guadeloupe, Mar. 2007, pp. 37-37. DOI: https://doi.org/10.1109/ICCGI.2007.62
W. Lee and S. J. Stolfo, "A framework for constructing features and models for intrusion detection systems," ACM Transactions on Information and System Security, vol. 3, no. 4, pp. 227-261, Nov. 2000. DOI: https://doi.org/10.1145/382912.382914
M. Koeman, J. Engel, J. Jansen, and L. Buydens, "Critical comparison of methods for fault diagnosis in metabolomics data," Scientific Reports, vol. 9, no. 1, Feb. 2019, Art. no. 1123. DOI: https://doi.org/10.1038/s41598-018-37494-7
H. Zou, T. Hastie, and R. Tibshirani, "Sparse Principal Component Analysis," Journal of Computational and Graphical Statistics, vol. 15, no. 2, pp. 265-286, Jun. 2006. DOI: https://doi.org/10.1198/106186006X113430
N. T. Pham, E. Foo, S. Suriadi, H. Jeffrey, and H. F. M. Lahza, "Improving performance of intrusion detection system using ensemble methods and feature selection," in Proceedings of the Australasian Computer Science Week Multiconference, New York, NY, USA, Jan. 2018, pp. 1-6. DOI: https://doi.org/10.1145/3167918.3167951
A. J. Malik, W. Shahzad, and F. A. Khan, "Network intrusion detection using hybrid binary PSO and random forests algorithm," Security and Communication Networks, vol. 8, no. 16, pp. 2646-2660, 2015. DOI: https://doi.org/10.1002/sec.508
Y. Zhong et al., "HELAD: A novel network anomaly detection model based on heterogeneous ensemble learning," Computer Networks, vol. 169, Mar. 2020, Art. no. 107049. DOI: https://doi.org/10.1016/j.comnet.2019.107049
F. Rezaei and A. Zahedi, "Dealing with Wormhole Attacks in Wireless Sensor Networks Through Discovering Separate Routes Between Nodes," Engineering, Technology & Applied Science Research, vol. 7, no. 4, pp. 1771-1774, Aug. 2017. DOI: https://doi.org/10.48084/etasr.1118
P. Ratadiya and R. Moorthy, "Spam filtering on forums: A synthetic oversampling based approach for imbalanced data classification," arXiv:1909.04826 [cs, stat], Sep. 2019, Accessed: Sep. 12, 2020. [Online]. Available: http://arxiv.org/abs/1909.04826.
P. More and M. P. Mishra, "Machine Learning for Cyber Threat Detection," International Journal of Advanced Trends in Computer Science and Engineering, vol. 9, no. 1.1, pp. 41-46, 2020. DOI: https://doi.org/10.30534/ijatcse/2020/0891.12020
M. Russinovich and T. Garnier, "Sysmon v11.11," Jul. 15, 2020. https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon (accessed Sep. 12, 2020).
Downloads
How to Cite
License
Copyright (c) 2020 Authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
