Comparative Evaluation of YOLO Models on an African Road Obstacles Dataset for Real-Time Obstacle Detection
Received: 29 September 2024 | Revised: 19 October 2024 and 31 October 2024 | Accepted: 9 November 2024 | Online: 26 November 2024
Corresponding author: Pison Mutabarura
Abstract
Public datasets are used to train road obstacle detection models, but they lack diverse and rare object classes found on African roads, negatively impacting the performance of models trained on them. Although attempts have been made to create custom datasets to train road obstacle detection models, they lack the unique challenges posed by African wildlife and livestock commonly encountered on African roads. This leads to poor performance of road obstacle detection systems in the African context. This study presents a custom dataset with rare African object classes and compares the performance of three YOLO models on it using mean Average Precision (mAP). The images were collected from multiple sources to ensure a wide range of scenarios. Offline data augmentation was applied to increase dataset diversity and simulate real-world road scenarios. The models were trained and evaluated, with YOLOv5 demonstrating superiority over the other two models, with an object detection accuracy of 94.68% mAP at an Intersection over Union (IoU) threshold of 0.5 with data augmentation. Offline data augmentation significantly improved all models' object detection accuracy, especially for YOLOv3. The results reveal the effectiveness of the custom dataset and highlight the importance of data augmentation in improving object detection.
Keywords:
African road obstacles, object detection, data augmentation, road obstacles, YOLOv3Downloads
References
R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, Jun. 2014, pp. 580–587.
R. Girshick, "Fast R-CNN." arXiv, Sep. 27, 2015, [Online].
S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137–1149, Jun. 2017.
P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, "Object Detection with Discriminatively Trained Part-Based Models," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 9, pp. 1627–1645, Sep. 2010.
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, Jun. 2016, pp. 779–788.
W. Liu et al., "SSD: Single Shot MultiBox Detector," in Computer Vision – ECCV 2016, Amsterdam, The Netherlands, 2016, pp. 21–37.
Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, "YOLOX: Exceeding YOLO Series in 2021." arXiv, Aug. 06, 2021.
J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, Jul. 2017, pp. 6517–6525.
K. Li, Y. Zhuang, J. Lai, and Y. Zeng, "PFYOLOv4: An Improved Small Object Pedestrian Detection Algorithm," IEEE Access, vol. 11, pp. 17197–17206, 2023.
A. Bochkovskiy, C. Y. Wang, and H. Y. M. Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection." arXiv, Apr. 23, 2020.
U. Nepal and H. Eslamiat, "Comparing YOLOv3, YOLOv4 and YOLOv5 for Autonomous Landing Spot Detection in Faulty UAVs," Sensors, vol. 22, no. 2, Jan. 2022, Art. no. 464.
Y. Zhang, Z. Guo, J. Wu, Y. Tian, H. Tang, and X. Guo, "Real-Time Vehicle Detection Based on Improved YOLO v5," Sustainability, vol. 14, no. 19, Jan. 2022, Art. no. 12274.
R. Rajamohanan and B. C. Latha, "An Optimized YOLO v5 Model for Tomato Leaf Disease Classification with Field Dataset," Engineering, Technology & Applied Science Research, vol. 13, no. 6, pp. 12033–12038, Dec. 2023.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The Pascal Visual Object Classes Challenge: A Retrospective," International Journal of Computer Vision, vol. 111, no. 1, pp. 98–136, Jan. 2015.
A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? The KITTI vision benchmark suite," in 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, Jun. 2012, pp. 3354–3361.
T. Y. Lin et al., "Microsoft COCO: Common Objects in Context," in Computer Vision – ECCV 2014, Zurich, Switzerland, 2014, pp. 740–755.
O. Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision, vol. 115, no. 3, pp. 211–252, Dec. 2015.
O. Zendel, M. Schorghuber, B. Rainer, M. Murschitz, and C. Beleznai, "Unifying Panoptic Segmentation for Autonomous Driving," in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, Jun. 2022, pp. 21319–21328.
D. Wu et al., "YOLOP: You Only Look Once for Panoptic Driving Perception," Machine Intelligence Research, vol. 19, no. 6, pp. 550–562, Dec. 2022.
H. L. Nguyen, D. T. Le, and H. H. Hoang, "Application of Synthetic Data on Object Detection Tasks," Engineering, Technology & Applied Science Research, vol. 14, no. 4, pp. 15695–15699, Aug. 2024.
W. Jiang, C. Song, H. Wang, M. Yu, and Y. Yan, "Obstacle Detection by Autonomous Vehicles: An Adaptive Neighborhood Search Radius Clustering Approach," Machines, vol. 11, no. 1, Jan. 2023, Art. no. 54.
F. Gao, C. Li, and B. Zhang, "A Dynamic Clustering Algorithm for Lidar Obstacle Detection of Autonomous Driving System," IEEE Sensors Journal, vol. 21, no. 22, Aug. 2021, Art. no. 25922–25930.
G. Al-Ρefai and M. Al-Ρefai, "Road object detection using Yolov3 and Kitti dataset," International Journal of Advanced Computer Science and Applications, vol. 11, no. 8, 2020.
A. Mumuni and F. Mumuni, "Data augmentation: A comprehensive survey of modern approaches," Array, vol. 16, Dec. 2022, Art. no. 100258.
P. Siripatthiti, "Data Augmentations for Improving Vision-Based Damage Detection : in Land Transport Infrastructure," M.S. Thesis, Kungliga Tekniska Högskolan (KTH), Stockholm, Sweden, 2023.
"Ultralytics | Revolutionizing the World of Vision AI." https://www.ultralytics.com/.
"OpenMMLab," GitHub. https://github.com/open-mmlab.
"openvinotoolkit/openvino." OpenVINOTM Toolkit, Nov. 15, 2024, [Online]. Available: https://github.com/openvinotoolkit/openvino.
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "RandAugment: Practical automated data augmentation with a reduced search space." arXiv, Nov. 14, 2019.
M. Patil, M. M. Patil, and S. Agrawal, "WGAN for Data Augmentation," in GANs for Data Augmentation in Healthcare, A. Solanki and M. Naved, Eds. Cham, Switzerland: Springer International Publishing, 2023, pp. 223–241.
G. B. Rajendran, U. M. Kumarasamy, C. Zarro, P. B. Divakarachari, and S. L. Ullo, "Land-Use and Land-Cover Classification Using a Human Group-Based Particle Swarm Optimization Algorithm with an LSTM Classifier on Hybrid Pre-Processing Remote-Sensing Images," Remote Sensing, vol. 12, no. 24, Jan. 2020, Art. no. 4135.
P. Sun et al., "Scalability in Perception for Autonomous Driving: Waymo Open Dataset," in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, Jun. 2020, pp. 2443–2451.
P. Sun et al., "Scalability in Perception for Autonomous Driving: Waymo Open Dataset." arXiv, May 12, 2020.
C. Sakaridis et al., "ACDC: The Adverse Conditions Dataset with Correspondences for Robust Semantic Driving Scene Perception." arXiv, Jun. 07, 2024.
M. Meyer and G. Kuschk, "Automotive Radar Dataset for Deep Learning Based 3D Object Detection," in 2019 16th European Radar Conference (EuRAD), Paris, France, Jul. 2019, pp. 129–132.
M. Sheeny, E. De Pellegrin, S. Mukherjee, A. Ahrabian, S. Wang, and A. Wallace, "RADIATE: A Radar Dataset for Automotive Perception in Bad Weather," in 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, May 2021, pp. 1–7.
J. Li, Y. Zhao, L. Gao, and F. Cui, "Compression of YOLOv3 via Block-Wise and Channel-Wise Pruning for Real-Time and Complicated Autonomous Driving Environment Sensing Applications," in 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, Jan. 2021, pp. 5107–5114.
C. N. Jaikishore et al., "Implementation of Deep Learning Algorithm on a Custom Dataset for Advanced Driver Assistance Systems Applications," Applied Sciences, vol. 12, no. 18, Jan. 2022, Art. no. 8927.
"Imageye - Image Downloader." https://www.imageye.net/.
"Open Images V7." https://storage.googleapis.com/openimages/web/index.html.
A. Menon, B. Omman, and A. S, "Pedestrian Counting Using Yolo V3," in 2021 International Conference on Innovative Trends in Information Technology (ICITIIT), Kottayam, India, Feb. 2021, pp. 1–9.
"Roboflow: Computer vision tools for developers and enterprises." https://roboflow.com/.
"figshare - credit for all your research." https://figshare.com/.
"YOLO: Real-Time Object Detection," Twitch. https://pjreddie.com/darknet/yolo/.
A. Bochkovskii, "AlexeyAB/darknet." Nov. 15, 2024, [Online]. Available: https://github.com/AlexeyAB/darknet.
J. S. Walia and K. Seemakurthy, "Optimized Custom Dataset for Efficient Detection of Underwater Trash," in Towards Autonomous Robotic Systems, Cambridge, UK, 2023, pp. 292–303.
A. Kuznetsova, T. Maleva, and V. Soloviev, "YOLOv5 versus YOLOv3 for Apple Detection," in Cyber-Physical Systems: Modelling and Intelligent Control, A. G. Kravets, A. A. Bolshakov, and M. Shcherbakov, Eds. Cham, Switzerland: Springer International Publishing, 2021, pp. 349–358.
O. Kıvrak and M. Z. Gürbüz, "Performance Comparison of YOLOv3,YOLOv4 and YOLOv5 algorithms : A Case Study for Poultry Recognition," Avrupa Bilim ve Teknoloji Dergisi, no. 38, pp. 392–397, Aug. 2022.
S. S. A. Zaidi, M. S. Ansari, A. Aslam, N. Kanwal, M. Asghar, and B. Lee, "A survey of modern deep learning based object detection models," Digital Signal Processing, vol. 126, Jun. 2022, Art. no. 103514.
A. Ben Atitallah, Y. Said, M. A. Ben Atitallah, M. Albekairi, K. Kaaniche, and S. Boubaker, "An effective obstacle detection system using deep learning advantages to aid blind and visually impaired navigation," Ain Shams Engineering Journal, vol. 15, no. 2, Feb. 2024, Art. no. 102387.
Downloads
How to Cite
License
Copyright (c) 2024 Pison Mutabarura, Nicasio Muchuka, Davies Segera
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
