A Constrained Multi-Objective Learning Algorithm for Feed-Forward Neural Network Classifiers

M. Njah, R. El Hamdi


This paper proposes a new approach to address the optimal design of a Feed-forward Neural Network (FNN) based classifier. The originality of the proposed methodology, called CMOA, lie in the use of a new constraint handling technique based on a self-adaptive penalty procedure in order to direct the entire search effort towards finding only Pareto optimal solutions that are acceptable. Neurons and connections of the FNN Classifier are dynamically built during the learning process. The approach includes differential evolution to create new individuals and then keeps only the non-dominated ones as the basis for the next generation. The designed FNN Classifier is applied to six binary classification benchmark problems, obtained from the UCI repository, and results indicated the advantages of the proposed approach over other existing multi-objective evolutionary neural networks classifiers reported recently in the literature.


FNN Classifier; Constrained Multi-Objective Optimization; Pareto Dominance Criterion; Differential Evolution

Full Text:



C. Aggarwal, Data Classification: Algorithms and Applications, 1st Edition. Chapman and Hall/CRC, 2014

R. Ramli, H. Arof, F. Ibrahim, M. Y. I. Idris, A. S. M. Khairuddin, “Classification of Eyelid Position and Eyeball Movement Using EEG Signals”, Malaysian Journal of Computer Science. Vol. 28, No. 1, pp. pp. 28-45, 2015

S. M. A. Kalaiarasi, S. Gopala, C. Ali, T. Jason, “Artificial Neural Network Tree Approach In Data Mining”, Malaysian Journal of Computer Science, Vol. 20, No. 1, pp. 51-62, 2007

V. Chaitali, B. Nikita, M. Darshana, “A survey on various classification techniques for clinical decision support system” International Journal of Computer Applications, Vol. 116, No. 23, pp. 14–17, 2015

A. Azzini, A. G. B. Tettamanzi, “Evolutionary ANNS: a state of the art survey”, Intelligenza Artificiale,Vol. 5, pp. 19–35, 2011

Y. Jin, B. Sendhoff, “Pareto-based multi-objective machine learning: An overview and case studies”, IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, Vol. 38, No. 3, pp. 397–415, 2008

T. Fawcett, “An introduction to roc analysis”, Pattern Recognition Letters, Vol. 27, No. 3, pp. 861–874, 2006

J. Fieldsend, S. Singh, “Pareto evolutionary neural networks”, IEEE Transactions on Neural Networks, Vol. 16, No. 2, pp. 338–354, 2005

S. K. Sharma, P. Chandra, “Constructive neural networks: A review”, International Journal of Engineering Science and Technology, Vol. 2, No. 12, pp. 7847–7855, 2010

A. Engelbrecht, “A new pruning heuristic based on variance analysis of sensitivity information”, IEEE Transactions on Neural Networks, Vol. 11, No. 6, pp. 1386–1399, 2011

H. Chen, Diversity and Regularization in Neural Network Ensembles, Ph.D. Thesis, School of Computer Science University of Birmingham, 2008

H. A. Abbass, R. Sarker, C. Newton, “PDE: a pareto-frontier differential evolution approach for multi-objective optimization problems”, 2001 Congress on Evolutionary Computation, Seoul, South Korea, pp. 971–978, 2001

H. A. Abbass, “A memetic Pareto evolutionary approach to artificial neural networks”, Australian Joint Conference on Artificial Intelligence, pp. 1–12, 2001

H. A. Abbass, “An evolutionary artificial neural network approach for breast cancer diagnosis”, Artificial Intelligence in Medicine, Vol. 25, pp. 265–281, 2002

J. Teo, “Evolutionary Multi-Objective Optimization For Automatic Synthesis Of Artificial Neural Network Robot Controllers”, Malaysian Journal of Computer Science, Vol. 18, No. 2, pp. 54-62, 2005.

K. Krawiec, W. Jaskowski, M. Szubert, “Multi-objective convolutional learning for face labeling”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3451-3459, 2015

S. Wiegand, C. Igel, “Evolutionary multi-objective optimization of neural networks for face detection”, International Journal of Computational Intelligence and Applications, Vol. 4, No. 3, pp. 237–253), 2004

K. Deb, A. Pratab, S. Agarwal, T. Meyarivan, “A fast and elitist multi-objective genetic algorithm: NSGA2”, IEEE Transactions on Evolutionary Computation, Vol. 6, No. 2, pp. 182–197, 2002

N. Q. Sultan, M. S. Siti, M. Z. Azlan, “Multi-objective hybrid evolutionary algorithms for radial basis function neural network design”, Knowledge-Based Systems. Vol. 27, pp. 475–497, 2012

N. Q. Sultan, M. S. Siti, Z. M. H. Siti, “Memetic multiobjective particle swarm optimization based radial basis function network for classification problems”, Information Sciences, Vol. 239, pp. 165–190, 2013

I. Kokshenev, A. Padua Braga, “An efficient multi-objective learning algorithm for RBF neural network”, Neurocomputing, Vol. 73, pp. 2799–2808, 2010

M. Cruz-Ramirez, J. C. Fernandez, J. Sanchez-Monedero, C. Hervas-Martinez, “Memetic Pareto differential evolutionary artificial neural networks to determine growth multi-classes in predictive microbiology”, Evolutionary Intelligence, Vol. 3, pp. 187–199, 2010

C. Igel, M. Husken, “Improving the Rprop learning algorithm”, Second International ICSC Symposium on Neural Computation (NC 2000, pp. 115–121, 2000

P. Ducange, B. Lazzerini, F. Marcelloni, “Multi-objective genetic fuzzy classifiers for imbalanced and cost-sensitive datasets”, Soft Computing, Vol. 14, pp. 713–728, 2010

C. Ferri, J. Hernndez-Orallo, R. Modroiu, “An experimental comparison of performance measures for classification”, Pattern Recognition Letters, Vol. 30, pp. 27–38, 2009

P. L. Bartlett, “The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network”, IEEE Transactions on Information Theory, Vol. 44, No. 2, pp. 525–536, 1998

G. Dreyfus, J. Martinez, M. Samuelides, M. B. Cordon, F. Badran, S. Thiria, L. Herault, Réseaux de neurones: méthodologie et application, Eyrolles, 2004

R. Storn, “Differential evolution research-Trends and open questions”, Studies in Computational Intelligence, Vol. 143, pp. 1-31, 2008

J. C. Fernandez Caballero, F. C. H. José Martinez, P. A. Gutiérrez, “Sensitivity versus accuracy in multiclass problems using memetic Pareto evolutionary neural networks”, IEEE Transactions on Neural Networks, Vol. 21, No. 5, pp. 750–770, 2010

eISSN: 1792-8036     pISSN: 2241-4487