Software Vulnerability Fuzz Testing: A Mutation-Selection Optimization Systematic Review
Received: 14 April 2024 | Revised: 6 May 2024 | Accepted: 14 May 2024 | Online: 2 August 2024
Corresponding author: Asia Othman Aljahdali
Abstract
As software vulnerabilities can cause cybersecurity threats and have severe consequences, it is necessary to develop effective techniques to discover such vulnerabilities. Fuzzing is one of the most widely employed approaches that has been adapted for software testing. The mutation-based fuzzing approach is currently the most popular. The state-of-the-art American Fuzzy Lop (AFL) selects mutations randomly and lacks knowledge of mutation operations that are more helpful in a particular stage. This study performs a systematic review to identify and analyze existing approaches that optimize the selection of mutation operations. The main contributions of this work are to draw attention to the importance of mutation operator selection, identify optimization algorithms for mutation operator selection, and investigate their impact on fuzzing testing in terms of code coverage and finding new vulnerabilities. The investigation shows the effectiveness and advantages of optimizing the selection of mutation operations to achieve higher code coverage and find more vulnerabilities.
Keywords:
Software testing, Software security, Fuzz testing, Vulnerabilities, Mutation operator selectionDownloads
References
M. N. A. Khan, A. M. Mirza, R. A. Wagan, M. Shahid, and I. Saleem, "A Literature Review on Software Testing Techniques for Smartphone Applications," Engineering, Technology & Applied Science Research, vol. 10, no. 6, pp. 6578–6583, Dec. 2020. DOI: https://doi.org/10.48084/etasr.3844
W. Alkaberi and F. Assiri, "Predicting the Number of Software Faults using Deep Learning," Engineering, Technology & Applied Science Research, vol. 14, no. 2, pp. 13222–13231, Apr. 2024. DOI: https://doi.org/10.48084/etasr.6798
D. Song et al., "PeriScope: An Effective Probing and Fuzzing Framework for the Hardware-OS Boundary," in Proceedings 2019 Network and Distributed System Security Symposium, San Diego, CA, USA, 2019. DOI: https://doi.org/10.14722/ndss.2019.23176
Y. Zheng, A. Davanian, H. Yin, C. Song, H. Zhu, and L. Sun, "FIRM-AFL: High-Throughput Greybox Fuzzing of IoT Firmware via Augmented Process Emulation," presented at the 28th USENIX Security Symposium (USENIX Security 19), Santa Clara, CA, USA, 2019, pp. 1099–1114.
S. Gorbunov and A. Rosenbloom, "AutoFuzz: Automated Network Protocol Fuzzing Framework," International Journal of Computer Science and Network Security, vol. 10, no. 8, pp. 239–245, 2010.
X. Zhu, S. Wen, S. Camtepe, and Y. Xiang, "Fuzzing: A Survey for Roadmap," ACM Computing Surveys, vol. 54, no. 11s, Jun. 2022, Art. no. 230. DOI: https://doi.org/10.1145/3512345
H. Liang, X. Pei, X. Jia, W. Shen, and J. Zhang, "Fuzzing: State of the Art," IEEE Transactions on Reliability, vol. 67, no. 3, pp. 1199–1218, Sep. 2018. DOI: https://doi.org/10.1109/TR.2018.2834476
M. Zalewski, "American Fuzzy Lop: A Security Oriented Fuzzer." Google, May 18, 2024, [Online]. Available: https://github.com/google/AFL.
C. Chen, B. Cui, J. Ma, R. Wu, J. Guo, and W. Liu, "A systematic review of fuzzing techniques," Computers & Security, vol. 75, pp. 118–137, Jun. 2018. DOI: https://doi.org/10.1016/j.cose.2018.02.002
V. J. M. Manès et al., "The Art, Science, and Engineering of Fuzzing: A Survey," IEEE Transactions on Software Engineering, vol. 47, no. 11, pp. 2312–2331, Nov. 2021. DOI: https://doi.org/10.1109/TSE.2019.2946563
J. Li, B. Zhao, and C. Zhang, "Fuzzing: a survey," Cybersecurity, vol. 1, no. 1, Jun. 2018, Art. no. 6. DOI: https://doi.org/10.1186/s42400-018-0002-y
Y. Wang, P. Jia, L. Liu, C. Huang, and Z. Liu, "A systematic review of fuzzing based on machine learning techniques," PLOS ONE, vol. 15, no. 8, 2020, Art. no. e0237749. DOI: https://doi.org/10.1371/journal.pone.0237749
C. Zhang, Y. Wang, and L. Wang, "Firmware Fuzzing: The State of the Art," in Proceedings of the 12th Asia-Pacific Symposium on Internetware, Singapore, Apr. 2021, pp. 110–115. DOI: https://doi.org/10.1145/3457913.3457934
K. Dewey, J. Roesch, and B. Hardekopf, "Language fuzzing using constraint logic programming," in Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, Vasteras, Sweden, Jun. 2014, pp. 725–730. DOI: https://doi.org/10.1145/2642937.2642963
K. Dewey, J. Roesch, and B. Hardekopf, "Fuzzing the Rust Typechecker Using CLP (T)," in 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), Lincoln, NE, USA, Aug. 2015, pp. 482–493. DOI: https://doi.org/10.1109/ASE.2015.65
M. Zalewski, "AFL: American Fuzzy Lop." [Online]. Available: https://lcamtuf.coredump.cx/afl/.
V. Ganesh, T. Leek, and M. Rinard, "Taint-based directed whitebox fuzzing," in 2009 IEEE 31st International Conference on Software Engineering, Vancouver, Canada, May 2009, pp. 474–484. DOI: https://doi.org/10.1109/ICSE.2009.5070546
C. Cadar, D. Dunbar, and D. Engler, "KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs," in 8th USENIX Symposium on Operating Systems Design and Implementation, San Diego, CA, USA, Dec. 2008, pp. 209–224.
P. Godefroid, M. Y. Levin, and D. Molnar, "Automated Whitebox Fuzz Testing," Network and Distributed System Security (NDSS), vol. 8, pp. 151–166, 2008.
D. Aitel, "An Introduction to SPIKE, the Fuzzer Creation Kit."
S. Hocevar, "Zzuf: Application Fuzzer." [Online]. Available: https://github.com/samhocevar/zzuf.
C. Chen, H. Xu, and B. Cui, "PSOFuzzer: A Target-Oriented Software Vulnerability Detection Technology Based on Particle Swarm Optimization," Applied Sciences, vol. 11, no. 3, Jan. 2021, Art. no. 1095. DOI: https://doi.org/10.3390/app11031095
X. Zhao, H. Qu, W. Lv, S. Li, and J. Xu, "MooFuzz: Many-Objective Optimization Seed Schedule for Fuzzer," Mathematics, vol. 9, no. 3, Jan. 2021, Art. no. 205. DOI: https://doi.org/10.3390/math9030205
Y. Fu, S. Tong, X. Guo, L. Cheng, Y. Zhang, and D. Feng, "Improving the Effectiveness of Grey-box Fuzzing By Extracting Program Information," in 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), Guangzhou, China, Dec. 2020, pp. 434–441. DOI: https://doi.org/10.1109/TrustCom50675.2020.00066
R. Shakya and A. Rahman, "A preliminary taxonomy of techniques used in software fuzzing," in Proceedings of the 7th Symposium on Hot Topics in the Science of Security, Jun. 2020. DOI: https://doi.org/10.1145/3384217.3384219
B. Kitchenham, "Procedures for Performing Systematic Reviews," Keele University, Technical Report TR/SE-0401, Jul. 2004.
S. Karamcheti, G. Mann, and D. Rosenberg, "Adaptive Grey-Box Fuzz-Testing with Thompson Sampling," in Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, Toronto, Canada, Jan. 2018, pp. 37–47. DOI: https://doi.org/10.1145/3270101.3270108
P. Godefroid, H. Peleg, and R. Singh, "Learn&Fuzz: Machine learning for input fuzzing," in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), Urbana, IL, USA, Oct. 2017, pp. 50–59. DOI: https://doi.org/10.1109/ASE.2017.8115618
D. She, K. Pei, D. Epstein, J. Yang, B. Ray, and S. Jana, "NEUZZ: Efficient Fuzzing with Neural Program Smoothing," in 2019 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, 2019, pp. 803–817. DOI: https://doi.org/10.1109/SP.2019.00052
X. Xie et al., "DeepHunter: a coverage-guided fuzz testing framework for deep neural networks," in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Beijing, China, Apr. 2019, pp. 146–157. DOI: https://doi.org/10.1145/3293882.3330579
K. Böttinger, P. Godefroid, and R. Singh, "Deep Reinforcement Fuzzing," in 2018 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA, 2018, pp. 116–122. DOI: https://doi.org/10.1109/SPW.2018.00026
Z. Zhang, B. Cui, and C. Chen, "Reinforcement Learning-Based Fuzzing Technology," in Innovative Mobile and Internet Services in Ubiquitous Computing, 2021, pp. 244–253. DOI: https://doi.org/10.1007/978-3-030-50399-4_24
H. Xu, B. Cui, and C. Chen, "Fuzzing with Multi-dimensional Control of Mutation Strategy," in Innovative Mobile and Internet Services in Ubiquitous Computing, Asan, Korea (South), 2022, pp. 276–284. DOI: https://doi.org/10.1007/978-3-030-79728-7_27
C. Lyu et al., "MOPT: Optimized Mutation Scheduling for Fuzzers," presented at the 28th USENIX Security Symposium (USENIX Security 19), Santa Clara, CA, USA, 2019, pp. 1949–1966.
M. E. Garbelini, C. Wang, and S. Chattopadhyay, "Greyhound: Directed Greybox Wi-Fi Fuzzing," IEEE Transactions on Dependable and Secure Computing, vol. 19, no. 2, pp. 817–834, Aug. 2022. DOI: https://doi.org/10.1109/TDSC.2020.3014624
X. Zhu and M. Böhme, "Regression Greybox Fuzzing," in Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, Aug. 2021, pp. 2169–2182. DOI: https://doi.org/10.1145/3460120.3484596
S. Rawat, V. Jain, A. Kumar, L. Cojocar, C. Giuffrida, and H. Bos, "VUzzer: Application-aware Evolutionary Fuzzing," in NDSS Symposium 2017, San Diego, CA, USA, Feb. 2017. DOI: https://doi.org/10.14722/ndss.2017.23404
T. Yue, Y. Tang, B. Yu, P. Wang, and E. Wang, "LearnAFL: Greybox Fuzzing With Knowledge Enhancement," IEEE Access, vol. 7, pp. 117029–117043, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2936235
C. Lemieux, R. Padhye, K. Sen, and D. Song, "PerfFuzz: automatically generating pathological inputs," in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, Amsterdam, Netherlands, Apr. 2018, pp. 254–265. DOI: https://doi.org/10.1145/3213846.3213874
L. Sun, X. Li, H. Qu, and X. Zhang, "AFLTurbo: Speed up Path Discovery for Greybox Fuzzing," in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), Oct. 2020, pp. 81–91. DOI: https://doi.org/10.1109/ISSRE5003.2020.00017
I. Nikolić, R. Mantu, S. Shen, and P. Saxena, "Refined Grey-Box Fuzzing with Sivo," in Detection of Intrusions and Malware, and Vulnerability Assessment, 2021, pp. 106–129. DOI: https://doi.org/10.1007/978-3-030-80825-9_6
J. Deng, X. Zhu, X. Xiao, S. Wen, Q. Li, and S. Xia, "Fuzzing With Optimized Grammar-Aware Mutation Strategies," IEEE Access, vol. 9, pp. 95061–95071, 2021. DOI: https://doi.org/10.1109/ACCESS.2021.3093904
V. T. Pham, M. Böhme, A. E. Santosa, A. R. Căciulescu, and A. Roychoudhury, "Smart Greybox Fuzzing," IEEE Transactions on Software Engineering, vol. 47, no. 9, pp. 1980–1997, Sep. 2019.
L. Situ, L. Wang, X. Li, L. Guan, W. Zhang, and P. Liu, "Energy Distribution Matters in Greybox Fuzzing," in 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), Montreal, Canada, May 2019, pp. 270–271. DOI: https://doi.org/10.1109/ICSE-Companion.2019.00109
C. Lemieux and K. Sen, "FairFuzz: a targeted mutation strategy for increasing greybox fuzz testing coverage," in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, Montpellier, France, Jun. 2018, pp. 475–485. DOI: https://doi.org/10.1145/3238147.3238176
U. Kargén and N. Shahmehri, "Speeding Up Bug Finding using Focused Fuzzing," in Proceedings of the 13th International Conference on Availability, Reliability and Security, Hamburg, Germany, May 2018, pp. 1–10. DOI: https://doi.org/10.1145/3230833.3230867
Y. Hu, W. Yang, B. Cui, X. Zhou, Z. Mao, and Y. Wang, "Fuzzing Method Based on Selection Mutation of Partition Weight Table for 5G Core Network NGAP Protocol," in Innovative Mobile and Internet Services in Ubiquitous Computing, Asan, Korea (South), 2022, pp. 144–155. DOI: https://doi.org/10.1007/978-3-030-79728-7_15
P. Chen and H. Chen, "Angora: Efficient Fuzzing by Principled Search," in 2018 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, May 2018, pp. 711–725. DOI: https://doi.org/10.1109/SP.2018.00046
X. Wang, C. Hu, R. Ma, D. Tian, and J. He, "CMFuzz: context-aware adaptive mutation for fuzzers," Empirical Software Engineering, vol. 26, no. 1, Jan. 2021, Art. no. 10. DOI: https://doi.org/10.1007/s10664-020-09927-3
V. Jain, S. Rawat, C. Giuffrida, and H. Bos, "TIFF: Using Input Type Inference To Improve Fuzzing," in Proceedings of the 34th Annual Computer Security Applications Conference, San Juan, PR, USA, Sep. 2018, pp. 505–517. DOI: https://doi.org/10.1145/3274694.3274746
Z. Cai, H. Wang, and X. Qin, "A Heuristic Guided Optimized Strategy for Non-Deterministic Mutation," in Proceedings of the 3rd International Conference on Computer Science and Application Engineering, Sanya, China, Oct. 2019. DOI: https://doi.org/10.1145/3331453.3361295
R. Padhye, C. Lemieux, and K. Sen, "JQF: coverage-guided property-based testing in Java," in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, Beijing, China, Jul. 2019, pp. 398–401. DOI: https://doi.org/10.1145/3293882.3339002
L. J. Moukahal, M. Zulkernine, and M. Soukup, "Vulnerability-Oriented Fuzz Testing for Connected Autonomous Vehicle Systems," IEEE Transactions on Reliability, vol. 70, no. 4, pp. 1422–1437, Oct. 2021. DOI: https://doi.org/10.1109/TR.2021.3112538
J. Liang et al., "DeepFuzzer: Accelerated Deep Greybox Fuzzing," IEEE Transactions on Dependable and Secure Computing, vol. 18, no. 6, pp. 2675–2688, Aug. 2021.
J. Wang, B. Chen, L. Wei, and Y. Liu, "Superion: Grammar-Aware Greybox Fuzzing," in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), Montreal, Canada, May 2019, pp. 724–735. DOI: https://doi.org/10.1109/ICSE.2019.00081
Downloads
How to Cite
License
Copyright (c) 2024 Fatmah Yousef Assiri, Asia Othman Aljahdali
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
