A Novel Two-Stage Selection of Feature Subsets in Machine Learning
Abstract
In feature subset selection the variable selection procedure selects a subset of the most relevant features. Filter and wrapper methods are categories of variable selection methods. Feature subsets are similar to data pre-processing and are applied to reduce feature dimensions in a very large dataset. In this paper, in order to deal with this kind of problems, the selection of feature subset methods depending on the fitness evaluation of the classifier is introduced to alleviate the classification task and to progress the classification performance. To curtail the dimensions of the feature space, a novel approach for selecting optimal features on two-stage selection of feature subsets (TSFS) method is done, both theoretically and experimentally. The results of this method include improvements in the performance measures like efficiency, accuracy, and scalability of machine learning algorithms. Comparison of the proposed method is made with known relevant methods using benchmark databases. The proposed method performs better than the earlier hybrid feature selection methodologies discussed in relevant works, regarding classifiers’ accuracy and error.
Keywords:
dimensionality reduction, feature subset selection, filter method, hybrid method, variable selection, wrapper methodDownloads
References
M. Dash, H. Liu, “Feature selection for classification”, Intelligent Data Analysis, Vol. 1, No. 1-4, pp. 131–156, 1997 DOI: https://doi.org/10.1016/S1088-467X(97)00008-5
R. Kohavi, G. H. John, “Wrappers for feature subset selection”, Artificial Intelligence, Vol. 97, No. 1-2, pp. 273–324, 1997 DOI: https://doi.org/10.1016/S0004-3702(97)00043-X
F. R. Kamala, P. R. J. Thangaiah, “A proposed two phase hybrid feature selection method using backward Elimination and PSO”, International Journal of Applied Engineering Research, Vol. 11, No. 1, pp. 77–83, 2016
M. Dash, H. Liu. “Consistency-based search in feature selection”, Artificial Intelligence, Vol. 151, No. 1-2, pp. 155–176, 2003 DOI: https://doi.org/10.1016/S0004-3702(03)00079-1
J. Kennedy, R. C. Eberhart, “Particle swarm optimization”, IEEE International Conference on Neural Networks, Perth, Australia, November27-December 1, 1995
N. Holden, A. A. Freitas, “A Hybrid PSO/ACO Algorithm for Discovering Classification Rules in Data Mining”, Journal of Artificial Evolution and Applications, Vol. 2008, ArticleID 316145, pp. 1-11, 2008 DOI: https://doi.org/10.1155/2008/316145
Indriyani, W. Gunawan, A. Rakhmadi, “Filter-Wrapper Approach to Feature Selection Using PSO-GA for Arabic Document Classification with Naive Bayes Multinomial”, IOSR Journal of Computer Engineering, Vol. 17, No. 6, pp. 45-51, 2015
W. Shang, H. Huang, H. Zhu, Y. Lin, Y. Qu, Z. Wang, “A Novel feature selection algorithm for text categorization”, Expert Systems with Applications, Vol. 33, No. 1, pp. 1-5, 2007 DOI: https://doi.org/10.1016/j.eswa.2006.04.001
H. Lu, J. Chen, K. Yan, Q. Jin, Z. Gao, “A hybrid feature selection algorithm for gene expression data classification”, Neurocomputing, Vol. 256, pp. 56-62, 2017 DOI: https://doi.org/10.1016/j.neucom.2016.07.080
A. S. Ghareb, A. A. Bakar, A. R. Hamdan, “Hybrid feature selection based on enhanced genetic algorithm for text categorization”, Expert Systems with Applications, Vol. 49, pp. 31-47, 2016 DOI: https://doi.org/10.1016/j.eswa.2015.12.004
A. B. Brahim, M. Limam, “A hybrid feature selection method based on instance learning and cooperative subset search”, Pattern Recognition Letters, Vol. 69, pp. 28-34, 2016 DOI: https://doi.org/10.1016/j.patrec.2015.10.005
K. Pearson, “On a criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling”, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, Vol. 50, No. 302, pp. 157–175, 1900 DOI: https://doi.org/10.1080/14786440009463897
P. E. Greenwood, M. S. Nikulin, A Guide to Chi-squared Testing, John Wiley & Sons, 1996
H. O. Lancaster, The Chi-squared Distribution, John Wiley & Sons, 1969
Y. Yang, J. O. Pedersen, “A comparative study on feature selection in text categorization”, in: Proceedings of the 14th international conference on machine learning(ICML) San Francisco, USA, pp. 412–420, Morgan Kaufmann Publishers, 1997
I. H. Witten, E. Frank, Data Mining Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, 2005
A. Jacobson, A. D. Milman, D. M. Kammen, “Letting the (energy) Gini out of the bottle: Lorenz curves of cumulative electricity consumption and Gini coefficients as metrics of energy distribution and equity”, Energy Policy, Vol. 33, No. 14, pp. 1825-1832, 2007 DOI: https://doi.org/10.1016/j.enpol.2004.02.017
L. Breiman, J. Friedman, R. Olshen, C. Stone, Classification and Regression Trees, Chapman and Hall, 1984
D. Wettschereck, D. Aha, W. T. Mohri, “A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms”, Artificial Intelligence Review, Vol. 11, No. 1-5, pp. 273-314, 1997 DOI: https://doi.org/10.1023/A:1006593614256
UCI Machine Learning Repository, University of California, available at: https://archive.ics.uci.edu/ml/index.php
V. S. Stehman, “Selecting and interpreting measures of thematic classification accuracy”, Remote Sensing of Environment, Vol. 62, No. 1, pp. 77–89, 1997 DOI: https://doi.org/10.1016/S0034-4257(97)00083-7
J. R. Landis, G. G. Koch, “The measurement of observer agreement for categorical data”, Biometrics, Vol. 33, No. 1, pp. 159–174, 1977 DOI: https://doi.org/10.2307/2529310
J. L. Fleiss, Statistical Methods for Rates and Proportions, John Wiley, 1981
J. Abellan, C. J. Mantas, J. G. Castellano, S. Moral-Garcia, “Increasing diversity in random forest learning algorithm via imprecise probabilities”, Expert Systems With Applications, Vol. 97, pp. 228–243, 2018 DOI: https://doi.org/10.1016/j.eswa.2017.12.029
K. J. Wang, A. M. Adrian, K. H. Chen, K. M. Wang, “An improved electromagnetism-like mechanism algorithm and its application to the prediction of diabetes mellitus”, Journal of Biomedical Informatics, Vol. 54, pp. 220–229, 2015 DOI: https://doi.org/10.1016/j.jbi.2015.02.001
Q. Tu, X. Chen, X. Liu, “Multi-strategy ensemble grey wolf optimizer and its application to feature selection”, Applied Soft Computing, Vol. 76, pp. 16–30, 2019 DOI: https://doi.org/10.1016/j.asoc.2018.11.047
F. Wang, J. Liang, “An efficient feature selection algorithm for hybrid data”, Neurocomputing, Vol. 193, pp. 33–41, 2016 DOI: https://doi.org/10.1016/j.neucom.2016.01.056
C. Li, H. Li. “A Survey of Distance Metrics for Nominal Attributes”, Journal of Software, Vol. 5, No. 11, pp. 1262-1269, 2010 DOI: https://doi.org/10.4304/jsw.5.11.1262-1269
Y. Huang, P. J. McCullagh, N. D. Black, “An optimization of ReliefF for classification in large datasets”, Knowledge Engineering, Vol. 68, No. 11, pp. 1348-1356, 2009 DOI: https://doi.org/10.1016/j.datak.2009.07.011
M. Nekkaa, D. Boughaci, “A memetic algorithm with support vector machine for feature selection and classification”, Memetic Computing. Vol. 7, No. 1, pp. 59–73, 2015 DOI: https://doi.org/10.1007/s12293-015-0153-2
L. T. Kueti, N. Tsopze, C. Mbiethieu, E. Mephu-Nguifo, L. P. Fotso, “Using Boolean factors for the construction of an artificial neural network”, International Journal of General Systems, Vol. 47, No. 8, pp. 849-868, 2018 DOI: https://doi.org/10.1080/03081079.2018.1524893
A. E. Hegazy, M. A. Makhlouf, G. S. El-Tawel, “Improved salp swarm algorithm for feature selection”, Journal of King Saud University – Computer and Information Sciences, 2018
B. Xue, Particle Swarm Optimisation for Feature Selection in Classification, PhD Thesis, Victoria University of Wellington, 2014 DOI: https://doi.org/10.1109/CEC.2014.6900472
C. Pascoal, M. R. Oliveira, A. Pacheco, R. Valadas, “Theoretical evaluation of feature selection methods based on mutual information”, Neurocomputing, Vol. 226, pp. 168–181, 2017 DOI: https://doi.org/10.1016/j.neucom.2016.11.047
Downloads
How to Cite
License
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
