A Hybrid Meta-Learner Technique for Credit Scoring of Banks’ Customers
Abstract
Financial institutions are exposed to credit risk due to issuance of consumer loans. Thus, developing reliable credit scoring systems is very crucial for them. Since, machine learning techniques have demonstrated their applicability and merit, they have been extensively used in credit scoring literature. Recent studies concentrating on hybrid models through merging various machine learning algorithms have revealed compelling results. There are two types of hybridization methods namely traditional and ensemble methods. This study combines both of them and comes up with a hybrid meta-learner model. The structure of the model is based on the traditional hybrid model of ‘classification + clustering’ in which the stacking ensemble method is employed in the classification part. Moreover, this paper compares several versions of the proposed hybrid model by using various combinations of classification and clustering algorithms. Hence, it helps us to identify which hybrid model can achieve the best performance for credit scoring purposes. Using four real-life credit datasets, the experimental results show that the model of (KNN-NN-SVMPSO)-(DL)-(DBSCAN) delivers the highest prediction accuracy and the lowest error rates.
Keywords:
credit scoring, hybrid machine learning models, stacking, deep learningDownloads
References
D. J. Hand, W. E. Henley, “Statistical classification methods in consumer credit scoring: a review”, Journal of the Royal Statistical Society: Series A (Statistics in Society), Vol. 160, No. 3, pp. 523-541, 1997 DOI: https://doi.org/10.1111/j.1467-985X.1997.00078.x
C. R. Abrahams, M. Zhang, Fair lending compliance: Intelligence and implications for credit risk management, John Wiley & Sons, 2008
E. Rosenberg, A. Gleit, “Quantitative methods in credit management: a survey”, Operations Research, Vol. 42, No. 4, pp. 589-613, 1994 DOI: https://doi.org/10.1287/opre.42.4.589
L. C. Thomas, D. B. Edelman, J. N. Crook, Credit scoring and its applications, SIAM, 2002 DOI: https://doi.org/10.1137/1.9780898718317
D. West, “Neural network credit scoring models”, Computers & Operations Research, Vol. 27, No. 11-12, pp. 1131-1152, 2000 DOI: https://doi.org/10.1016/S0305-0548(99)00149-5
R. C. Lacher, P. K. Coats, S. C. Sharma, L. F. Fant, “A neural network for classifying the financial health of a firm”, European Journal of Operational Research, Vol. 85, No. 1, pp. 53-65, 1995 DOI: https://doi.org/10.1016/0377-2217(93)E0274-2
T.-S. Lee, I.-F. Chen, “A two-stage hybrid credit scoring model using artificial neural networks and multivariate adaptive regression splines”, Expert Systems with Applications, Vol. 28, No. 4, pp. 743-752, 2005 DOI: https://doi.org/10.1016/j.eswa.2004.12.031
H. Abdou, J. Pointon, A. El-Masry, “Neural nets versus conventional techniques in credit scoring in Egyptian banking”, Expert Systems with Applications, Vol. 35, No. 3, pp. 1275-1292, 2008 DOI: https://doi.org/10.1016/j.eswa.2007.08.030
M.-C. Chen, S.-H. Huang, “Credit scoring and rejected instances reassigning through evolutionary computation techniques”, Expert Systems with Applications, Vol. 24, No. 4, pp. 433-441, 2003 DOI: https://doi.org/10.1016/S0957-4174(02)00191-4
V. S. Desai, J. N. Crook, G. A. Overstreet, “A comparison of neural networks and linear scoring models in the credit union environment”, European Journal of Operational Research, Vol. 95, No. 1, pp. 24-37, 1996 DOI: https://doi.org/10.1016/0377-2217(95)00246-4
A. Lahsasna, R. N. Ainon, T. Y. Wah, “Enhancement of transparency and accuracy of credit scoring models through genetic fuzzy classifier”, Maejo International Journal of Science and Technology, Vol. 4, No. 1, pp. 136-158, 2010
T.-S. Lee, C.-C. Chiu, C.-J. Lu, I.-F. Chen, “Credit scoring using the hybrid neural discriminant technique”, Expert Systems with applications, Vol. 23, No. 3, pp. 245-254, 2002 DOI: https://doi.org/10.1016/S0957-4174(02)00044-1
M.-C. Tsai, S.-P. Lin, C.-C. Cheng, Y.-P. Lin, “The consumer loan default predicting model–An application of DEA–DA and neural network”, Expert Systems with applications, Vol. 36, No. 9, pp. 11682-11690, 2009 DOI: https://doi.org/10.1016/j.eswa.2009.03.009
J. N. Crook, D. B. Edelman, L. C. Thomas, “Recent developments in consumer credit risk assessment”, European Journal of Operational Research, Vol. 183, No. 3, pp. 1447-1465, 2007 DOI: https://doi.org/10.1016/j.ejor.2006.09.100
Z. Huang, H. Chen, C.-J. Hsu, W.-H. Chen, S. Wu, “Credit rating analysis with support vector machines and neural networks: a market comparative study”, Decision Support Systems, Vol. 37, No. 4, pp. 543-558, 2004 DOI: https://doi.org/10.1016/S0167-9236(03)00086-1
C.-S. Ong, J.-J. Huang, G.-H. Tzeng, “Building credit scoring models using genetic programming”, Expert Systems with Applications, Vol. 29, No. 1, pp. 41-47, 2005 DOI: https://doi.org/10.1016/j.eswa.2005.01.003
N.-C. Hsieh, “Hybrid mining approach in the design of credit scoring models”, Expert Systems with Applications, Vol. 28, No. 4, pp. 655-665, 2005 DOI: https://doi.org/10.1016/j.eswa.2004.12.022
A. Jain, A. M. Kumar, “Hybrid neural network models for hydrologic time series forecasting”, Applied Soft Computing, Vol. 7, No. 2, pp. 585-592, 2007 DOI: https://doi.org/10.1016/j.asoc.2006.03.002
H. Kim, K. Shin, “A hybrid approach based on neural networks and genetic algorithms for detecting temporal patterns in stock markets”, Applied Soft Computing, Vol. 7, No. 2, pp. 569-576, 2007 DOI: https://doi.org/10.1016/j.asoc.2006.03.004
R. Malhotra, D. K. Malhotra, “Differentiating between good credits and bad credits using neuro-fuzzy systems”, European Journal of Operational research, Vol. 136, No. 1, pp. 190-211, 2002 DOI: https://doi.org/10.1016/S0377-2217(01)00052-2
J. Huysmans, B. Baesens, J. Vanthienen, T. Van Gestel, “Failure prediction with self organizing maps”, Expert Systems with Applications, Vol. 30, No. 3, pp. 479-487, 2006 DOI: https://doi.org/10.1016/j.eswa.2005.10.005
C. Tsai, M. Chen, “Credit rating by hybrid machine learning techniques”, Applied Soft Computing, Vol. 10, No. 2, pp. 374-380, 2010 DOI: https://doi.org/10.1016/j.asoc.2009.08.003
I. Brown, C. Mues, “An experimental comparison of classification algorithms for imbalanced credit scoring data sets”, Expert Systems with Applications, Vol. 39, No. 3, pp. 3446-3453, 2012 DOI: https://doi.org/10.1016/j.eswa.2011.09.033
G. Wang, J. Ma, L. Huang, K. Xu, “Two credit scoring models based on dual strategy ensemble trees”, Knowledge-Based Systems, Vol. 26, pp. 61-68, 2012 DOI: https://doi.org/10.1016/j.knosys.2011.06.020
T. Harris, “Credit scoring using the clustered support vector machine”, Expert Systems with Applications, Vol. 42, No. 2, pp. 741-750, 2015 DOI: https://doi.org/10.1016/j.eswa.2014.08.029
H. Xiao, Z. Xiao, Y. Wang, “Ensemble classification based on supervised clustering for credit scoring”, Applied Soft Computing, Vol. 43, pp. 73-86, 2016 DOI: https://doi.org/10.1016/j.asoc.2016.02.022
J. Shi, B. Xu, “Credit Scoring by Fuzzy Support Vector Machines with a Novel Membership Function”, Journal of Risk and Financial Management, Vol. 9, No. 4, pp. 13, 2016 DOI: https://doi.org/10.3390/jrfm9040013
M. Ala'raj, M. F. Abbod, “A new hybrid ensemble credit scoring model based on classifiers consensus system approach”, Expert Systems with Applications, Vol. 64, pp. 36-55, 2016 DOI: https://doi.org/10.1016/j.eswa.2016.07.017
M. J. Lenard, G. R. Madey, P. Alam, “The design and validation of a hybrid information system for the auditor’s going concern decision”, Journal of Management Information Systems, Vol. 14, No. 4, pp. 219-237, 1998 DOI: https://doi.org/10.1080/07421222.1998.11518192
T. G. Dietterich, “Ensemble learning”, The handbook of brain theory and neural networks, Vol. 2, pp. 110-125, 2002
M. Tavana, K. Puranam, Handbook of Research on Organizational Transformations through Big Data Analytics, IGI Global, 2014 DOI: https://doi.org/10.4018/978-1-4666-7272-7
F. Anifowose, J. Labadin, A. Abdulraheem, “Improving the prediction of petroleum reservoir characterization with a stacked generalization ensemble model of support vector machines”, Applied Soft Computing, Vol. 26, pp. 483-496, 2015 DOI: https://doi.org/10.1016/j.asoc.2014.10.017
J. Kittler, M. Hatef, R. P. Duin, J. Matas, “On combining classifiers”, IEEE transactions on pattern analysis and machine intelligence, Vol. 20, No. 3, pp. 226-239, 1998 DOI: https://doi.org/10.1109/34.667881
D. Opitz, R. Maclin, “Popular ensemble methods: An empirical study”, Journal of Artificial Intelligence Research, Vol. 11, pp. 169-198, 1999 DOI: https://doi.org/10.1613/jair.614
M. Pal, “Ensemble learning with decision tree for remote sensing classification”, World Academy of Science, Engineering and Technology, Vol. 1, No. 12, pp. 3839-3841, 2007
D. H. Wolpert, “Stacked generalization”, Neural networks, Vol. 5, No. 2, pp. 241-259, 1992 DOI: https://doi.org/10.1016/S0893-6080(05)80023-1
J. Sill, G. Takacs, L. Mackey, D. Lin, “Feature-weighted linear stacking”, arXiv:0911.0460, 2009
M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents”, Tenth International Conference on Machine Learning pp. 330-337, 1993 DOI: https://doi.org/10.1016/B978-1-55860-307-3.50049-6
. Breiman, “Stacked regressions”, Machine learning, Vol. 24, No. 1, pp. 49-64, 1996 DOI: https://doi.org/10.1007/BF00117832
M. Ozay, F. T. Y. Vural, “A new fuzzy stacked generalization technique and analysis of its performance”, arXiv:1204.0171, 2012
P. Smyth, D. Wolpert, “Linearly combining density estimators via stacking”, Machine Learning, Vol. 36, No. 1-2, pp. 59-83, 1999 DOI: https://doi.org/10.1023/A:1007511322260
T. M. Mitchell, Machine learning. 1997, Burr Ridge, IL: McGraw Hill, 1997
A. K. Jain, M. N. Murty, P. J. Flynn, “Data clustering: a review”, ACM Computing Surveys (CSUR), Vol. 31, No. 3, pp. 264-323, 1999 DOI: https://doi.org/10.1145/331499.331504
G. E. Hinton, S. Osindero, Y.-W. Teh, “A fast learning algorithm for deep belief nets”, Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006 DOI: https://doi.org/10.1162/neco.2006.18.7.1527
I. Goodfellow, Q. V. Le, A. M. Saxe, H. Lee, A. Y. Ng, “Measuring invariances in deep networks”, 23rd Annual Conference on Neural Information Processing Systems, pp. 646-654, 2009
D. C. Ciresan, U. Meier, L. M. Gambardella, J. Schmidhuber, “Deep, big, simple neural nets for handwritten digit recognition”, Neural Computation, Vol. 22, No. 12, pp. 3207-3220, 2010 DOI: https://doi.org/10.1162/NECO_a_00052
Y. Bengio, “Practical recommendations for gradient-based training of deep architectures”, arXiv:1206.5533, 2012 DOI: https://doi.org/10.1007/978-3-642-35289-8_26
Y. Bengio, A. Courville, P. Vincent, “Representation learning: A review and new perspectives”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 8, pp. 1798-1828, 2013 DOI: https://doi.org/10.1109/TPAMI.2013.50
A. Candel, V. Parmar, E. LeDell, A. Arora, Deep Learning with H2O, H2O Inc., 2016
S. Ramaswamy, R. Rastogi, K. Shim, “Efficient algorithms for mining outliers from large data sets”, ACM SIGMOD International Conference On Management Of Data, pp. 427-438, 2000 DOI: https://doi.org/10.1145/335191.335437
C. Cortes, V. Vapnik, “Support-vector networks”, Machine Learning, Vol. 20, No. 3, pp. 273-297, 1995 DOI: https://doi.org/10.1007/BF00994018
S. Li, W. Shiue, M. Huang, “The evaluation of consumer loans using support vector machines”, Expert Systems with Applications, Vol. 30, No. 4, pp. 772-782, 2006 DOI: https://doi.org/10.1016/j.eswa.2005.07.041
R. Eberhart, J. Kennedy, “A new optimizer using particle swarm theory”, Sixth International Symposium on Micro Machine and Human Science pp. 39-43, 1995
J. F. Kennedy, J. Kennedy, R. C. Eberhart, Y. Shi, Swarm intelligence, Morgan Kaufmann, 2001
M. Ester, H. Kriegel, J. Sander, X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise”, Second International Conference on Knowledge Discovery and Data Mining, pp. 226-231, 1996
C. F. Tsai, Y. F. Hsu, “A Meta‐learning Framework for Bankruptcy Prediction”, Journal of Forecasting, Vol. 32, No. 2, pp. 167-179, 2013 DOI: https://doi.org/10.1002/for.1264
F. S. C. Analide, “Information asset analysis: credit scoring and credit suggestion”, International Journal of Electronic Business, Vol. 9, No. 3, pp. 203-218, 2011 DOI: https://doi.org/10.1504/IJEB.2011.042542
Y. Peng, G. Kou, Y. Shi, Z. Chen, “A multi-criteria convex quadratic programming model for credit data analysis”, Decision Support Systems, Vol. 44, No. 4, pp. 1016-1030, 2008 DOI: https://doi.org/10.1016/j.dss.2007.12.001
C. Tsai, C. Hung, “Modeling credit scoring using neural network ensembles”, Kybernetes, Vol. 43, No. 7, pp. 1114-1123, 2014 DOI: https://doi.org/10.1108/K-01-2014-0016
A. Marcano-Cedeno, A. Marin-De-La-Barcena, J. Jimenez-Trillo, J. Pinuela, D. Andina, “Artificial metaplasticity neural network applied to credit scoring”, International Journal of Neural Systems, Vol. 21, No. 04, pp. 311-317, 2011 DOI: https://doi.org/10.1142/S0129065711002857
P. Somol, B. Baesens, P. Pudil, J. Vanthienen, “Filter‐versus wrapper‐based feature selection for credit scoring”, International Journal of Intelligent Systems, Vol. 20, No. 10, pp. 985-999, 2005 DOI: https://doi.org/10.1002/int.20103
C. Tsai, J. Wu, “Using neural network ensembles for bankruptcy prediction and credit scoring”, Expert Systems with Applications, Vol. 34, No. 4, pp. 2639-2649, 2008 DOI: https://doi.org/10.1016/j.eswa.2007.05.019
D. Zhang, X. Zhou, S. C. Leung, J. Zheng, “Vertical bagging decision trees model for credit scoring”, Expert Systems with Applications, Vol. 37, No. 12, pp. 7838-7843, 2010 DOI: https://doi.org/10.1016/j.eswa.2010.04.054
Z. Qi, B. Wang, Y. Tian, P. Zhang, “When Ensemble Learning Meets Deep Learning: a New Deep Support Vector Machine for Classification”, Knowledge-Based Systems, Vol. 107, pp. 54-60, 2016 DOI: https://doi.org/10.1016/j.knosys.2016.05.055
S. Lin, K. Ying, S. Chen, Z. Lee, “Particle swarm optimization for parameter determination and feature selection of support vector machines”, Expert Systems with Applications, Vol. 35, No. 4, pp. 1817-1824, 2008 DOI: https://doi.org/10.1016/j.eswa.2007.08.088
D. Martens, B. Baesens, T. Van Gestel, J. Vanthienen, “Comprehensible credit scoring models using rule extraction from support vector machines”, European Journal of Operational Research, Vol. 183, No. 3, pp. 1466-1476, 2007 DOI: https://doi.org/10.1016/j.ejor.2006.04.051
S. Luo, B. Cheng, C. Hsieh, “Prediction model building with clustering-launched classification and support vector machines in credit scoring”, Expert Systems with Applications, Vol. 36, No. 4, pp. 7562-7566, 2009 DOI: https://doi.org/10.1016/j.eswa.2008.09.028
C. F. Tsai, “Financial decision support using neural networks and support vector machines”, Expert Systems, Vol. 25, No. 4, pp. 380-393, 2008 DOI: https://doi.org/10.1111/j.1468-0394.2008.00449.x
J. Huang, G. Tzeng, C. Ong, “Two-stage genetic programming (2SGP) for the credit scoring model”, Applied Mathematics and Computation, Vol. 174, No. 2, pp. 1039-1053, 2006 DOI: https://doi.org/10.1016/j.amc.2005.05.027
H. Van Sang, N. H. Nam, N. D. Nhan, “A novel credit scoring prediction model based on Feature Selection approach and parallel random forest”, Indian Journal of Science and Technology, Vol. 9, No. 20, 2016 DOI: https://doi.org/10.17485/ijst/2016/v9i20/92299
B. Baesens, T. Van Gestel, S. Viaene, M. Stepanova, J. Suykens, J. Vanthienen, “Benchmarking state-of-the-art classification algorithms for credit scoring”, Journal of the Operational Research Society, Vol. 54, No. 6, pp. 627-635, 2003 DOI: https://doi.org/10.1057/palgrave.jors.2601545
C. Tsai, “Feature selection in bankruptcy prediction”, Knowledge-Based Systems, Vol. 22, No. 2, pp. 120-127, 2009 DOI: https://doi.org/10.1016/j.knosys.2008.08.002
C. Huang, M. Chen, C. Wang, “Credit scoring with a data mining approach based on support vector machines”, Expert Systems with Applications, Vol. 33, No. 4, pp. 847-856, 2007 DOI: https://doi.org/10.1016/j.eswa.2006.07.007
Y. Ping, L. Yongheng, “Neighborhood rough set and SVM based hybrid credit scoring classifier”, Expert Systems with Applications, Vol. 38, No. 9, pp. 11300-11304, 2011 DOI: https://doi.org/10.1016/j.eswa.2011.02.179
F. Chen, F. Li, “Combination of feature selection approaches with SVM in credit scoring”, Expert Systems with Applications, Vol. 37, No. 7, pp. 4902-4909, 2010 DOI: https://doi.org/10.1016/j.eswa.2009.12.025
F. Hoffmann, B. Baesens, C. Mues, T. Van Gestel, J. Vanthienen, “Inferring descriptive and approximate fuzzy rules for credit scoring using evolutionary algorithms”, European Journal of Operational Research, Vol. 177, No. 1, pp. 540-555, 2007 DOI: https://doi.org/10.1016/j.ejor.2005.09.044
X. Liu, H. Fu, W. Lin, “A modified support vector machine model for credit scoring”, International Journal of Computational Intelligence Systems, Vol. 3, No. 6, pp. 797-804, 2010 DOI: https://doi.org/10.1080/18756891.2010.9727742
L. Nanni, A. Lumini, “An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring”, Expert Systems with Applications, Vol. 36, No. 2, pp. 3028-3033, 2009 DOI: https://doi.org/10.1016/j.eswa.2008.01.018
Y. Lan, D. Janssens, G. Chen, G. Wets, “Improving associative classification by incorporating novel interestingness measures”, Expert Systems with Applications, Vol. 31, No. 1, pp. 184-192, 2006 DOI: https://doi.org/10.1016/j.eswa.2005.09.015
S. Oreski, D. Oreski, G. Oreski, “Hybrid system with genetic algorithm and artificial neural networks and its application to retail credit risk assessment”, Expert Systems with Applications, Vol. 39, No. 16, pp. 12605-12617, 2012 DOI: https://doi.org/10.1016/j.eswa.2012.05.023
Downloads
How to Cite
License
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
