A Hybrid Meta-Learner Technique for Credit Scoring of Banks’ Customers

A. G. Armaki, M. F. Fallah, M. Alborzi, A. Mohammadzadeh


Financial institutions are exposed to credit risk due to issuance of consumer loans. Thus, developing reliable credit scoring systems is very crucial for them. Since, machine learning techniques have demonstrated their applicability and merit, they have been extensively used in credit scoring literature. Recent studies concentrating on hybrid models through merging various machine learning algorithms have revealed compelling results. There are two types of hybridization methods namely traditional and ensemble methods. This study combines both of them and comes up with a hybrid meta-learner model. The structure of the model is based on the traditional hybrid model of ‘classification + clustering’ in which the stacking ensemble method is employed in the classification part. Moreover, this paper compares several versions of the proposed hybrid model by using various combinations of classification and clustering algorithms. Hence, it helps us to identify which hybrid model can achieve the best performance for credit scoring purposes. Using four real-life credit datasets, the experimental results show that the model of (KNN-NN-SVMPSO)-(DL)-(DBSCAN) delivers the highest prediction accuracy and the lowest error rates.


credit scoring; hybrid machine learning models; stacking; deep learning

Full Text:



D. J. Hand, W. E. Henley, “Statistical classification methods in consumer credit scoring: a review”, Journal of the Royal Statistical Society: Series A (Statistics in Society), Vol. 160, No. 3, pp. 523-541, 1997

C. R. Abrahams, M. Zhang, Fair lending compliance: Intelligence and implications for credit risk management, John Wiley & Sons, 2008

E. Rosenberg, A. Gleit, “Quantitative methods in credit management: a survey”, Operations Research, Vol. 42, No. 4, pp. 589-613, 1994

L. C. Thomas, D. B. Edelman, J. N. Crook, Credit scoring and its applications, SIAM, 2002

D. West, “Neural network credit scoring models”, Computers & Operations Research, Vol. 27, No. 11-12, pp. 1131-1152, 2000

R. C. Lacher, P. K. Coats, S. C. Sharma, L. F. Fant, “A neural network for classifying the financial health of a firm”, European Journal of Operational Research, Vol. 85, No. 1, pp. 53-65, 1995

T.-S. Lee, I.-F. Chen, “A two-stage hybrid credit scoring model using artificial neural networks and multivariate adaptive regression splines”, Expert Systems with Applications, Vol. 28, No. 4, pp. 743-752, 2005

H. Abdou, J. Pointon, A. El-Masry, “Neural nets versus conventional techniques in credit scoring in Egyptian banking”, Expert Systems with Applications, Vol. 35, No. 3, pp. 1275-1292, 2008

M.-C. Chen, S.-H. Huang, “Credit scoring and rejected instances reassigning through evolutionary computation techniques”, Expert Systems with Applications, Vol. 24, No. 4, pp. 433-441, 2003

V. S. Desai, J. N. Crook, G. A. Overstreet, “A comparison of neural networks and linear scoring models in the credit union environment”, European Journal of Operational Research, Vol. 95, No. 1, pp. 24-37, 1996

A. Lahsasna, R. N. Ainon, T. Y. Wah, “Enhancement of transparency and accuracy of credit scoring models through genetic fuzzy classifier”, Maejo International Journal of Science and Technology, Vol. 4, No. 1, pp. 136-158, 2010

T.-S. Lee, C.-C. Chiu, C.-J. Lu, I.-F. Chen, “Credit scoring using the hybrid neural discriminant technique”, Expert Systems with applications, Vol. 23, No. 3, pp. 245-254, 2002

M.-C. Tsai, S.-P. Lin, C.-C. Cheng, Y.-P. Lin, “The consumer loan default predicting model–An application of DEA–DA and neural network”, Expert Systems with applications, Vol. 36, No. 9, pp. 11682-11690, 2009

J. N. Crook, D. B. Edelman, L. C. Thomas, “Recent developments in consumer credit risk assessment”, European Journal of Operational Research, Vol. 183, No. 3, pp. 1447-1465, 2007

Z. Huang, H. Chen, C.-J. Hsu, W.-H. Chen, S. Wu, “Credit rating analysis with support vector machines and neural networks: a market comparative study”, Decision Support Systems, Vol. 37, No. 4, pp. 543-558, 2004

C.-S. Ong, J.-J. Huang, G.-H. Tzeng, “Building credit scoring models using genetic programming”, Expert Systems with Applications, Vol. 29, No. 1, pp. 41-47, 2005

N.-C. Hsieh, “Hybrid mining approach in the design of credit scoring models”, Expert Systems with Applications, Vol. 28, No. 4, pp. 655-665, 2005

A. Jain, A. M. Kumar, “Hybrid neural network models for hydrologic time series forecasting”, Applied Soft Computing, Vol. 7, No. 2, pp. 585-592, 2007

H. Kim, K. Shin, “A hybrid approach based on neural networks and genetic algorithms for detecting temporal patterns in stock markets”, Applied Soft Computing, Vol. 7, No. 2, pp. 569-576, 2007

R. Malhotra, D. K. Malhotra, “Differentiating between good credits and bad credits using neuro-fuzzy systems”, European Journal of Operational research, Vol. 136, No. 1, pp. 190-211, 2002

J. Huysmans, B. Baesens, J. Vanthienen, T. Van Gestel, “Failure prediction with self organizing maps”, Expert Systems with Applications, Vol. 30, No. 3, pp. 479-487, 2006

C. Tsai, M. Chen, “Credit rating by hybrid machine learning techniques”, Applied Soft Computing, Vol. 10, No. 2, pp. 374-380, 2010

I. Brown, C. Mues, “An experimental comparison of classification algorithms for imbalanced credit scoring data sets”, Expert Systems with Applications, Vol. 39, No. 3, pp. 3446-3453, 2012

G. Wang, J. Ma, L. Huang, K. Xu, “Two credit scoring models based on dual strategy ensemble trees”, Knowledge-Based Systems, Vol. 26, pp. 61-68, 2012

T. Harris, “Credit scoring using the clustered support vector machine”, Expert Systems with Applications, Vol. 42, No. 2, pp. 741-750, 2015

H. Xiao, Z. Xiao, Y. Wang, “Ensemble classification based on supervised clustering for credit scoring”, Applied Soft Computing, Vol. 43, pp. 73-86, 2016

J. Shi, B. Xu, “Credit Scoring by Fuzzy Support Vector Machines with a Novel Membership Function”, Journal of Risk and Financial Management, Vol. 9, No. 4, pp. 13, 2016

M. Ala'raj, M. F. Abbod, “A new hybrid ensemble credit scoring model based on classifiers consensus system approach”, Expert Systems with Applications, Vol. 64, pp. 36-55, 2016

M. J. Lenard, G. R. Madey, P. Alam, “The design and validation of a hybrid information system for the auditor’s going concern decision”, Journal of Management Information Systems, Vol. 14, No. 4, pp. 219-237, 1998

T. G. Dietterich, “Ensemble learning”, The handbook of brain theory and neural networks, Vol. 2, pp. 110-125, 2002

M. Tavana, K. Puranam, Handbook of Research on Organizational Transformations through Big Data Analytics, IGI Global, 2014

F. Anifowose, J. Labadin, A. Abdulraheem, “Improving the prediction of petroleum reservoir characterization with a stacked generalization ensemble model of support vector machines”, Applied Soft Computing, Vol. 26, pp. 483-496, 2015

J. Kittler, M. Hatef, R. P. Duin, J. Matas, “On combining classifiers”, IEEE transactions on pattern analysis and machine intelligence, Vol. 20, No. 3, pp. 226-239, 1998

D. Opitz, R. Maclin, “Popular ensemble methods: An empirical study”, Journal of Artificial Intelligence Research, Vol. 11, pp. 169-198, 1999

M. Pal, “Ensemble learning with decision tree for remote sensing classification”, World Academy of Science, Engineering and Technology, Vol. 1, No. 12, pp. 3839-3841, 2007

D. H. Wolpert, “Stacked generalization”, Neural networks, Vol. 5, No. 2, pp. 241-259, 1992

J. Sill, G. Takacs, L. Mackey, D. Lin, “Feature-weighted linear stacking”, arXiv:0911.0460, 2009

M. Tan, “Multi-agent reinforcement learning: Independent vs. cooperative agents”, Tenth International Conference on Machine Learning pp. 330-337, 1993

. Breiman, “Stacked regressions”, Machine learning, Vol. 24, No. 1, pp. 49-64, 1996

M. Ozay, F. T. Y. Vural, “A new fuzzy stacked generalization technique and analysis of its performance”, arXiv:1204.0171, 2012

P. Smyth, D. Wolpert, “Linearly combining density estimators via stacking”, Machine Learning, Vol. 36, No. 1-2, pp. 59-83, 1999

T. M. Mitchell, Machine learning. 1997, Burr Ridge, IL: McGraw Hill, 1997

A. K. Jain, M. N. Murty, P. J. Flynn, “Data clustering: a review”, ACM Computing Surveys (CSUR), Vol. 31, No. 3, pp. 264-323, 1999

G. E. Hinton, S. Osindero, Y.-W. Teh, “A fast learning algorithm for deep belief nets”, Neural Computation, Vol. 18, No. 7, pp. 1527-1554, 2006

I. Goodfellow, Q. V. Le, A. M. Saxe, H. Lee, A. Y. Ng, “Measuring invariances in deep networks”, 23rd Annual Conference on Neural Information Processing Systems, pp. 646-654, 2009

D. C. Ciresan, U. Meier, L. M. Gambardella, J. Schmidhuber, “Deep, big, simple neural nets for handwritten digit recognition”, Neural Computation, Vol. 22, No. 12, pp. 3207-3220, 2010

Y. Bengio, “Practical recommendations for gradient-based training of deep architectures”, arXiv:1206.5533, 2012

Y. Bengio, A. Courville, P. Vincent, “Representation learning: A review and new perspectives”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 8, pp. 1798-1828, 2013

A. Candel, V. Parmar, E. LeDell, A. Arora, Deep Learning with H2O, H2O Inc., 2016

S. Ramaswamy, R. Rastogi, K. Shim, “Efficient algorithms for mining outliers from large data sets”, ACM SIGMOD International Conference On Management Of Data, pp. 427-438, 2000

C. Cortes, V. Vapnik, “Support-vector networks”, Machine Learning, Vol. 20, No. 3, pp. 273-297, 1995

S. Li, W. Shiue, M. Huang, “The evaluation of consumer loans using support vector machines”, Expert Systems with Applications, Vol. 30, No. 4, pp. 772-782, 2006

R. Eberhart, J. Kennedy, “A new optimizer using particle swarm theory”, Sixth International Symposium on Micro Machine and Human Science pp. 39-43, 1995

J. F. Kennedy, J. Kennedy, R. C. Eberhart, Y. Shi, Swarm intelligence, Morgan Kaufmann, 2001

M. Ester, H. Kriegel, J. Sander, X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise”, Second International Conference on Knowledge Discovery and Data Mining, pp. 226-231, 1996

C. F. Tsai, Y. F. Hsu, “A Meta‐learning Framework for Bankruptcy Prediction”, Journal of Forecasting, Vol. 32, No. 2, pp. 167-179, 2013

F. S. C. Analide, “Information asset analysis: credit scoring and credit suggestion”, International Journal of Electronic Business, Vol. 9, No. 3, pp. 203-218, 2011

Y. Peng, G. Kou, Y. Shi, Z. Chen, “A multi-criteria convex quadratic programming model for credit data analysis”, Decision Support Systems, Vol. 44, No. 4, pp. 1016-1030, 2008

C. Tsai, C. Hung, “Modeling credit scoring using neural network ensembles”, Kybernetes, Vol. 43, No. 7, pp. 1114-1123, 2014

A. Marcano-Cedeno, A. Marin-De-La-Barcena, J. Jimenez-Trillo, J. Pinuela, D. Andina, “Artificial metaplasticity neural network applied to credit scoring”, International Journal of Neural Systems, Vol. 21, No. 04, pp. 311-317, 2011

P. Somol, B. Baesens, P. Pudil, J. Vanthienen, “Filter‐versus wrapper‐based feature selection for credit scoring”, International Journal of Intelligent Systems, Vol. 20, No. 10, pp. 985-999, 2005

C. Tsai, J. Wu, “Using neural network ensembles for bankruptcy prediction and credit scoring”, Expert Systems with Applications, Vol. 34, No. 4, pp. 2639-2649, 2008

D. Zhang, X. Zhou, S. C. Leung, J. Zheng, “Vertical bagging decision trees model for credit scoring”, Expert Systems with Applications, Vol. 37, No. 12, pp. 7838-7843, 2010

Z. Qi, B. Wang, Y. Tian, P. Zhang, “When Ensemble Learning Meets Deep Learning: a New Deep Support Vector Machine for Classification”, Knowledge-Based Systems, Vol. 107, pp. 54-60, 2016

S. Lin, K. Ying, S. Chen, Z. Lee, “Particle swarm optimization for parameter determination and feature selection of support vector machines”, Expert Systems with Applications, Vol. 35, No. 4, pp. 1817-1824, 2008

D. Martens, B. Baesens, T. Van Gestel, J. Vanthienen, “Comprehensible credit scoring models using rule extraction from support vector machines”, European Journal of Operational Research, Vol. 183, No. 3, pp. 1466-1476, 2007

S. Luo, B. Cheng, C. Hsieh, “Prediction model building with clustering-launched classification and support vector machines in credit scoring”, Expert Systems with Applications, Vol. 36, No. 4, pp. 7562-7566, 2009

C. F. Tsai, “Financial decision support using neural networks and support vector machines”, Expert Systems, Vol. 25, No. 4, pp. 380-393, 2008

J. Huang, G. Tzeng, C. Ong, “Two-stage genetic programming (2SGP) for the credit scoring model”, Applied Mathematics and Computation, Vol. 174, No. 2, pp. 1039-1053, 2006

H. Van Sang, N. H. Nam, N. D. Nhan, “A novel credit scoring prediction model based on Feature Selection approach and parallel random forest”, Indian Journal of Science and Technology, Vol. 9, No. 20, 2016

B. Baesens, T. Van Gestel, S. Viaene, M. Stepanova, J. Suykens, J. Vanthienen, “Benchmarking state-of-the-art classification algorithms for credit scoring”, Journal of the Operational Research Society, Vol. 54, No. 6, pp. 627-635, 2003

C. Tsai, “Feature selection in bankruptcy prediction”, Knowledge-Based Systems, Vol. 22, No. 2, pp. 120-127, 2009

C. Huang, M. Chen, C. Wang, “Credit scoring with a data mining approach based on support vector machines”, Expert Systems with Applications, Vol. 33, No. 4, pp. 847-856, 2007

Y. Ping, L. Yongheng, “Neighborhood rough set and SVM based hybrid credit scoring classifier”, Expert Systems with Applications, Vol. 38, No. 9, pp. 11300-11304, 2011

F. Chen, F. Li, “Combination of feature selection approaches with SVM in credit scoring”, Expert Systems with Applications, Vol. 37, No. 7, pp. 4902-4909, 2010

F. Hoffmann, B. Baesens, C. Mues, T. Van Gestel, J. Vanthienen, “Inferring descriptive and approximate fuzzy rules for credit scoring using evolutionary algorithms”, European Journal of Operational Research, Vol. 177, No. 1, pp. 540-555, 2007

X. Liu, H. Fu, W. Lin, “A modified support vector machine model for credit scoring”, International Journal of Computational Intelligence Systems, Vol. 3, No. 6, pp. 797-804, 2010

L. Nanni, A. Lumini, “An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring”, Expert Systems with Applications, Vol. 36, No. 2, pp. 3028-3033, 2009

Y. Lan, D. Janssens, G. Chen, G. Wets, “Improving associative classification by incorporating novel interestingness measures”, Expert Systems with Applications, Vol. 31, No. 1, pp. 184-192, 2006

S. Oreski, D. Oreski, G. Oreski, “Hybrid system with genetic algorithm and artificial neural networks and its application to retail credit risk assessment”, Expert Systems with Applications, Vol. 39, No. 16, pp. 12605-12617, 2012

eISSN: 1792-8036     pISSN: 2241-4487