A Novel Two-Stage Selection of Feature Subsets in Machine Learning

R. F. Kamala, P. R. J. Thangaiah


In feature subset selection the variable selection procedure selects a subset of the most relevant features. Filter and wrapper methods are categories of variable selection methods. Feature subsets are similar to data pre-processing and are applied to reduce feature dimensions in a very large dataset. In this paper, in order to deal with this kind of problems, the selection of feature subset methods depending on the fitness evaluation of the classifier is introduced to alleviate the classification task and to progress the classification performance. To curtail the dimensions of the feature space, a novel approach for selecting optimal features on two-stage selection of feature subsets (TSFS) method is done, both theoretically and experimentally. The results of this method include improvements in the performance measures like efficiency, accuracy, and scalability of machine learning algorithms. Comparison of the proposed method is made with known relevant methods using benchmark databases. The proposed method performs better than the earlier hybrid feature selection methodologies discussed in relevant works, regarding classifiers’ accuracy and error.


dimensionality reduction; feature subset selection; filter method; hybrid method; variable selection; wrapper method

Full Text:



M. Dash, H. Liu, “Feature selection for classification”, Intelligent Data Analysis, Vol. 1, No. 1-4, pp. 131–156, 1997

R. Kohavi, G. H. John, “Wrappers for feature subset selection”, Artificial Intelligence, Vol. 97, No. 1-2, pp. 273–324, 1997

F. R. Kamala, P. R. J. Thangaiah, “A proposed two phase hybrid feature selection method using backward Elimination and PSO”, International Journal of Applied Engineering Research, Vol. 11, No. 1, pp. 77–83, 2016

M. Dash, H. Liu. “Consistency-based search in feature selection”, Artificial Intelligence, Vol. 151, No. 1-2, pp. 155–176, 2003

J. Kennedy, R. C. Eberhart, “Particle swarm optimization”, IEEE International Conference on Neural Networks, Perth, Australia, November27-December 1, 1995

N. Holden, A. A. Freitas, “A Hybrid PSO/ACO Algorithm for Discovering Classification Rules in Data Mining”, Journal of Artificial Evolution and Applications, Vol. 2008, ArticleID 316145, pp. 1-11, 2008

Indriyani, W. Gunawan, A. Rakhmadi, “Filter-Wrapper Approach to Feature Selection Using PSO-GA for Arabic Document Classification with Naive Bayes Multinomial”, IOSR Journal of Computer Engineering, Vol. 17, No. 6, pp. 45-51, 2015

W. Shang, H. Huang, H. Zhu, Y. Lin, Y. Qu, Z. Wang, “A Novel feature selection algorithm for text categorization”, Expert Systems with Applications, Vol. 33, No. 1, pp. 1-5, 2007

H. Lu, J. Chen, K. Yan, Q. Jin, Z. Gao, “A hybrid feature selection algorithm for gene expression data classification”, Neurocomputing, Vol. 256, pp. 56-62, 2017

A. S. Ghareb, A. A. Bakar, A. R. Hamdan, “Hybrid feature selection based on enhanced genetic algorithm for text categorization”, Expert Systems with Applications, Vol. 49, pp. 31-47, 2016

A. B. Brahim, M. Limam, “A hybrid feature selection method based on instance learning and cooperative subset search”, Pattern Recognition Letters, Vol. 69, pp. 28-34, 2016

K. Pearson, “On a criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling”, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Vol. 50, No. 302, pp. 157–175, 1900

P. E. Greenwood, M. S. Nikulin, A Guide to Chi-squared Testing, John Wiley & Sons, 1996

H. O. Lancaster, The Chi-squared Distribution, John Wiley & Sons, 1969

Y. Yang, J. O. Pedersen, “A comparative study on feature selection in text categorization”, in: Proceedings of the 14th international conference on machine learning(ICML) San Francisco, USA, pp. 412–420, Morgan Kaufmann Publishers, 1997

I. H. Witten, E. Frank, Data Mining Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, 2005

A. Jacobson, A. D. Milman, D. M. Kammen, “Letting the (energy) Gini out of the bottle: Lorenz curves of cumulative electricity consumption and Gini coefficients as metrics of energy distribution and equity”, Energy Policy, Vol. 33, No. 14, pp. 1825-1832, 2007

L. Breiman, J. Friedman, R. Olshen, C. Stone, Classification and Regression Trees, Chapman and Hall, 1984

D. Wettschereck, D. Aha, W. T. Mohri, “A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms”, Artificial Intelligence Review, Vol. 11, No. 1-5, pp. 273-314, 1997

UCI Machine Learning Repository, University of California, available at: https://archive.ics.uci.edu/ml/index.php

V. S. Stehman, “Selecting and interpreting measures of thematic classification accuracy”, Remote Sensing of Environment, Vol. 62, No. 1, pp. 77–89, 1997

J. R. Landis, G. G. Koch, “The measurement of observer agreement for categorical data”, Biometrics, Vol. 33, No. 1, pp. 159–174, 1977

J. L. Fleiss, Statistical Methods for Rates and Proportions, John Wiley, 1981

J. Abellan, C. J. Mantas, J. G. Castellano, S. Moral-Garcia, “Increasing diversity in random forest learning algorithm via imprecise probabilities”, Expert Systems With Applications, Vol. 97, pp. 228–243, 2018

K. J. Wang, A. M. Adrian, K. H. Chen, K. M. Wang, “An improved electromagnetism-like mechanism algorithm and its application to the prediction of diabetes mellitus”, Journal of Biomedical Informatics, Vol. 54, pp. 220–229, 2015

Q. Tu, X. Chen, X. Liu, “Multi-strategy ensemble grey wolf optimizer and its application to feature selection”, Applied Soft Computing, Vol. 76, pp. 16–30, 2019

F. Wang, J. Liang, “An efficient feature selection algorithm for hybrid data”, Neurocomputing, Vol. 193, pp. 33–41, 2016

C. Li, H. Li. “A Survey of Distance Metrics for Nominal Attributes”, Journal of Software, Vol. 5, No. 11, pp. 1262-1269, 2010

Y. Huang, P. J. McCullagh, N. D. Black, “An optimization of ReliefF for classification in large datasets”, Knowledge Engineering, Vol. 68, No. 11, pp. 1348-1356, 2009

M. Nekkaa, D. Boughaci, “A memetic algorithm with support vector machine for feature selection and classification”, Memetic Computing. Vol. 7, No. 1, pp. 59–73, 2015

L. T. Kueti, N. Tsopze, C. Mbiethieu, E. Mephu-Nguifo, L. P. Fotso, “Using Boolean factors for the construction of an artificial neural network”, International Journal of General Systems, Vol. 47, No. 8, pp. 849-868, 2018

A. E. Hegazy, M. A. Makhlouf, G. S. El-Tawel, “Improved salp swarm algorithm for feature selection”, Journal of King Saud University – Computer and Information Sciences, 2018

B. Xue, Particle Swarm Optimisation for Feature Selection in Classification, PhD Thesis, Victoria University of Wellington, 2014

C. Pascoal, M. R. Oliveira, A. Pacheco, R. Valadas, “Theoretical evaluation of feature selection methods based on mutual information”, Neurocomputing, Vol. 226, pp. 168–181, 2017

eISSN: 1792-8036     pISSN: 2241-4487