Levy Enhanced Cross Entropy-based Optimized Training of Feedforward Neural Networks
Received: 11 July 2022 | Revised: 27 July 2022 | Accepted: 1 August 2022 | Online: 2 October 2022
Corresponding author: K. Pandya
Abstract
An Artificial Neural Network (ANN) is one of the most powerful tools to predict the behavior of a system with unforeseen data. The feedforward neural network is the simplest, yet most efficient topology that is widely used in computer industries. Training of feedforward ANNs is an integral part of an ANN-based system. Typically an ANN system has inherent non-linearity with multiple parameters like weights and biases that must be optimized simultaneously. To solve such a complex optimization problem, this paper proposes the Levy Enhanced Cross Entropy (LE-CE) method. It is a population-based meta-heuristic method. In each iteration, this method produces a "distribution" of prospective solutions and updates it by updating the parameters of the distribution to obtain the optimal solutions, unlike traditional meta-heuristic methods. As a result, it reduces the chances of getting trapped into local minima, which is the typical drawback of any AI method. To further improve the global exploration capability of the CE method, it is subjected to the Levy flight which consists of a large step length during intermediate iterations. The performance of the LE-CE method is compared with state-of-the-art optimization methods. The result shows the superiority of LE-CE. The statistical ANOVA test confirms that the proposed LE-CE is statistically superior to other algorithms.
Keywords:
artificial neural networks, cross entropy method, feedforward neural networks, Levy step, trainingDownloads
References
C.-J. Lin, C.-H. Chen, and C.-Y. Lee, "A self-adaptive quantum radial basis function network for classification applications," in International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), Budapest, Hungary, Jul. 2004, vol. 4, pp. 3263–3268.
S. Mirjalili, S. Z. Mohd Hashim, and H. Moradian Sardroudi, "Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm," Applied Mathematics and Computation, vol. 218, no. 22, pp. 11125–11137, Jul. 2012. DOI: https://doi.org/10.1016/j.amc.2012.04.069
K. Hornik, M. Stinchcombe, and H. White, "Multilayer feedforward networks are universal approximators," Neural Networks, vol. 2, no. 5, pp. 359–366, Jan. 1989. DOI: https://doi.org/10.1016/0893-6080(89)90020-8
J.-R. Zhang, J. Zhang, T.-M. Lok, and M. R. Lyu, "A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training," Applied Mathematics and Computation, vol. 185, no. 2, pp. 1026–1037, Feb. 2007. DOI: https://doi.org/10.1016/j.amc.2006.07.025
M. Gori and A. Tesi, "On the Problem of Local Minima in Backpropagation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 1, pp. 76–86, Jan. 1992. DOI: https://doi.org/10.1109/34.107014
L. V. Kantorovich, "Functional analysis and applied mathematics," Uspekhi Matematicheskikh Nauk, vol. 3, no. 6, pp. 89–185, 1948.
E. A. Ogbonnaya, E. M. Adigio, H. U. Ugwu, and M. C. Anumiri, "Advanced Gas turbine rotor shaft fault diagnosis using artificial neural network," International Journal of Engineering and Technology Innovation, vol. 3, no. 1, pp. 58–69, 2013.
R. Kaluri and P. Reddy CH, "Optimized feature extraction for precise sign gesture recognition using self-improved genetic algorithm," International Journal of Engineering and Technology Innovation, vol. 8, no. 1, pp. 25–37, 2018.
M. Njah and R. E. Hamdi, "A Constrained Multi-Objective Learning Algorithm for Feed-Forward Neural Network Classifiers," Engineering, Technology & Applied Science Research, vol. 7, no. 3, pp. 1685–1693, Jun. 2017. DOI: https://doi.org/10.48084/etasr.968
D. N. Truong and V. T. Bui, "Hybrid PSO-Optimized ANFIS-Based Model to Improve Dynamic Voltage Stability," Engineering, Technology & Applied Science Research, vol. 9, no. 4, pp. 4384–4388, Aug. 2019. DOI: https://doi.org/10.48084/etasr.2833
L. T. H. Nhung, T. T. Phung, H. M. V. Nguyen, T. N. Le, T. A. Nguyen, and T. D. Vo, "Load Shedding in Microgrids with Dual Neural Networks and AHP Algorithm," Engineering, Technology & Applied Science Research, vol. 12, no. 1, pp. 8090–8095, Feb. 2022. DOI: https://doi.org/10.48084/etasr.4652
E. Rashedi, H. Nezamabadi-pour, and S. Saryazdi, "GSA: A Gravitational Search Algorithm," Information Sciences, vol. 179, no. 13, pp. 2232–2248, Jun. 2009. DOI: https://doi.org/10.1016/j.ins.2009.03.004
R. Storn and K. Price, "Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces," Journal of Global Optimization, vol. 11, no. 4, pp. 341–359, Dec. 1997. DOI: https://doi.org/10.1023/A:1008202821328
J. Kennedy and R. Eberhart, "Particle swarm optimization," in International Conference on Neural Networks, Perth, WA, Australia, Dec. 1995, vol. 4, pp. 1942–1948.
D. Dabhi and K. Pandya, "Enhanced Velocity Differential Evolutionary Particle Swarm Optimization for Optimal Scheduling of a Distributed Energy Resources With Uncertain Scenarios," IEEE Access, vol. 8, pp. 27001–27017, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2970236
D. Dabhi and K. Pandya, "Uncertain Scenario Based MicroGrid Optimization via Hybrid Levy Particle Swarm Variable Neighborhood Search Optimization (HL_PS_VNSO)," IEEE Access, vol. 8, pp. 108782–108797, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2999935
R. Y. Rubinstein, "Optimization of computer simulation models with rare events," European Journal of Operational Research, vol. 99, no. 1, pp. 89–112, May 1997. DOI: https://doi.org/10.1016/S0377-2217(96)00385-2
C. T. Brown, L. S. Liebovitch, and R. Glendon, "Levy Flights in Dobe Ju/’hoansi Foraging Patterns," Human Ecology, vol. 35, no. 1, pp. 129–138, Feb. 2007. DOI: https://doi.org/10.1007/s10745-006-9083-4
X. S. Yang, "Random Walks and Levy Flights," in Nature-Inspired Metaheuristic Algorithms, 2nd ed., Cambridge, UK: Luniver Press, 2010, pp. 11–19.
D. H. Wolpert and W. G. Macready, "No free lunch theorems for optimization," IEEE Transactions on Evolutionary Computation, vol. 1, no. 1, pp. 67–82, Apr. 1997. DOI: https://doi.org/10.1109/4235.585893
R. A. Fisher, "The Use of Multiple Measurements in Taxonomic Problems," Annals of Eugenics, vol. 7, no. 2, pp. 179–188, 1936. DOI: https://doi.org/10.1111/j.1469-1809.1936.tb02137.x
"The Iris Dataset," Gist. https://gist.github.com/curran/a08a1080b88344
b0c8a7.
K. Thirunavukkarasu, A. S. Singh, P. Rai, and S. Gupta, "Classification of IRIS Dataset using Classification Based KNN Algorithm in Supervised Learning," in 4th International Conference on Computing Communication and Automation, Greater Noida, India, Dec. 2018, pp. 1–4. DOI: https://doi.org/10.1109/CCAA.2018.8777643
E. Ostertagova and O. Ostertag, "Methodology and Application of Oneway ANOVA," American Journal of Mechanical Engineering, vol. 1, no. 7, pp. 256–261, Jan. 2013.
H. Abdi and L. J. Williams, "Tukey’s Honestly Significant Difference (HSD) Test," in Encyclopedia of Research Design, Thousand Oaks, CA, USA: SAGE, 2010.
H. L. Harter, "Critical Values for Duncan’s New Multiple Range Test," Biometrics, vol. 16, no. 4, pp. 671–685, 1960. DOI: https://doi.org/10.2307/2527770
Downloads
How to Cite
License
Copyright (c) 2022 K. Pandya, D. Dabhi, P. Mochi, V. Rajput
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
