MediAnnote: A Framework for Collaborative Annotation and Retrieval of Chest X-Rays Using the Radiology Gamuts Ontology
Received: 8 November 2025 | Revised: 26 November 2025 | Accepted: 13 December 2025 | Online: 28 December 2025
Corresponding author: Mona Alshlowi
Abstract
Medical imaging is a cornerstone of healthcare, supporting disease diagnosis, treatment planning, and clinical research. The growing volume of medical imaging data has made image annotation an increasingly complex yet essential step in producing reliable, high-quality labeled datasets for diagnostic, educational, and research purposes. This work presents MediAnnote, an ontology-based framework for medical image annotation and retrieval that combines a deep learning component for pre-annotation with the Radiology Gamuts Ontology (RGO) within a collaborative environment. The framework enables image retrieval along with the corresponding findings and causes, enhancing both educational and clinical applications. MediAnnote outperformed existing annotation systems in a qualitative comparison incorporating all essential components. An experimental study involving three radiologists and the NIH Chest X-ray dataset showed that the model achieved a higher accuracy in disease prediction, with an F1-score of 0.54, an AUC of 0.75, a precision of 0.54, and a recall of 0.53, compared to individual radiologists. In addition, integrating a human-in-the-loop approach improved the precision of abnormality localization. The post-task survey showed high user satisfaction, with an overall mean score of 3.94 out of 5.
Keywords:
ontology, medical image annotation, semantic image retrieval, chest x-ray, semantic web, human-in-the-loop, deep learning, radiology AIDownloads
References
T. Hailemariam et al., "Chest X-ray predicts cases of pulmonary tuberculosis among women of reproductive age with acute respiratory symptoms: A multi-center cross-sectional study," Journal of Clinical Tuberculosis and Other Mycobacterial Diseases, vol. 32, Aug. 2023, Art. no. 100383. DOI: https://doi.org/10.1016/j.jctube.2023.100383
A. Q. Khan, S. Nayyar, S. Khalid, I. Taqi, and M. A. Khan, "Prevalence of Interstitial Lung Disease in Patients with Chronic Cough Taking Chest X-rays and CT Chest as Diagnostic Tools," Journal of Islamabad Medical & Dental College, vol. 13, no. 1, pp. 110–115, Apr. 2024. DOI: https://doi.org/10.35787/jimdc.v13i1.1073
A. Soni and A. Rai, "A systematic survey on deep learning techniques for chest disease detection using chest radiographs," Journal of Current Science and Technology, vol. 13, no. 2, pp. 267–295, July 2023. DOI: https://doi.org/10.59796/jcst.V13N2.2023.1744
S. Budd, E. C. Robinson, and B. Kainz, "A survey on active learning and human-in-the-loop deep learning for medical image analysis," Medical Image Analysis, vol. 71, July 2021, Art. no. 102062. DOI: https://doi.org/10.1016/j.media.2021.102062
A. Ryabtsev, R. Lederman, J. Sosna, and L. Joskowicz, "Streamlining the annotation process by radiologists of volumetric medical images with few-shot learning," International Journal of Computer Assisted Radiology and Surgery, vol. 20, no. 9, pp. 1863–1873, June 2025. DOI: https://doi.org/10.1007/s11548-025-03457-3
M. Napravnik, F. Hržić, S. Tschauner, and I. Štajduhar, "Building RadiologyNET: an unsupervised approach to annotating a large-scale multimodal medical database," BioData Mining, vol. 17, no. 1, July 2024, Art. no. 22. DOI: https://doi.org/10.1186/s13040-024-00373-1
A. Lawley, R. Hampson, K. Worrall, and G. Dobie, "Prescriptive Method for Optimizing Cost of Data Collection and Annotation in Machine Learning of Clinical Ultrasound," in 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Sydney, Australia, July 2023, pp. 1–4. DOI: https://doi.org/10.1109/EMBC40787.2023.10340858
Y. Nomura et al., "Performance changes due to differences among annotating radiologists for training data in computerized lesion detection," International Journal of Computer Assisted Radiology and Surgery, vol. 19, no. 8, pp. 1527–1536, Aug. 2024. DOI: https://doi.org/10.1007/s11548-024-03136-9
F. Galbusera and A. Cina, "Image annotation and curation in radiology: an overview for machine learning practitioners," European Radiology Experimental, vol. 8, no. 1, Feb. 2024, Art. no. 11. DOI: https://doi.org/10.1186/s41747-023-00408-y
K. Zhang et al., "Rep-GLS: Report-Guided Generalized Label Smoothing for Robust Disease Detection." arXiv, 2025.
"RadLex radiology lexicon," RAdiological Society of North America. https://www.rsna.org/practice-tools/data-tools-and-standards/radlex-radiology-lexicon.
"SNOMED International - Service Migration." https://static-web.snomedtools.org/html/migrate.html.
J. J. Budovec, C. A. Lam, and C. E. Kahn, "Informatics in Radiology: Radiology Gamuts Ontology: Differential Diagnosis for the Semantic Web," RadioGraphics, vol. 34, no. 1, pp. 254–264, Jan. 2014. DOI: https://doi.org/10.1148/rg.341135036
R. Chen, Z. Zhao, M. Yusufu, X. Shang, D. Shi, and M. He, "Choroidal Vessel Segmentation on Indocyanine Green Angiography Images via Human-in-the-Loop Labeling." arXiv, 2024.
Q. Zeng, Y. Xie, Z. Lu, and Y. Xia, "A human-in-the-loop method for pulmonary nodule detection in CT scans," Visual Intelligence, vol. 2, no. 1, July 2024, Art. no. 19. DOI: https://doi.org/10.1007/s44267-024-00052-z
M. Tounsi, E. Aram, A. T. Azar, A. Al-Khayyat, and I. K. Ibraheem, "A Comprehensive Review on Biomedical Image Classification using Deep Learning Models," Engineering, Technology & Applied Science Research, vol. 15, no. 1, pp. 19538–19545, Feb. 2025. DOI: https://doi.org/10.48084/etasr.8728
A. Amalki, K. Tatane, and A. Bouzit, "Deep Learning-Driven Ontology Learning: A Systematic Mapping Study," Engineering, Technology & Applied Science Research, vol. 15, no. 1, pp. 20085–20094, Feb. 2025. DOI: https://doi.org/10.48084/etasr.9431
F. M. Aldhafeeri, "Governing Artificial Intelligence in Radiology: A Systematic Review of Ethical, Legal, and Regulatory Frameworks," Diagnostics, vol. 15, no. 18, Sept. 2025, Art. no. 2300. DOI: https://doi.org/10.3390/diagnostics15182300
J. H. Lee, H. Hong, G. Nam, E. J. Hwang, and C. M. Park, "Effect of Human-AI Interaction on Detection of Malignant Lung Nodules on Chest Radiographs," Radiology, vol. 307, no. 5, June 2023, Art. no. e222976. DOI: https://doi.org/10.1148/radiol.222976
B. N. Patel et al., "Human–machine partnership with artificial intelligence for chest radiograph diagnosis," npj Digital Medicine, vol. 2, no. 1, Nov. 2019, Art. no. 111. DOI: https://doi.org/10.1038/s41746-019-0189-7
A. Patil et al., "Efficient quality control of whole slide pathology images with human-in-the-loop training," Journal of Pathology Informatics, vol. 14, 2023, Art. no. 100306. DOI: https://doi.org/10.1016/j.jpi.2023.100306
H. Wang, Q. Jin, S. Li, S. Liu, M. Wang, and Z. Song, "A comprehensive survey on deep active learning in medical image analysis," Medical Image Analysis, vol. 95, July 2024, Art. no. 103201. DOI: https://doi.org/10.1016/j.media.2024.103201
B. C. Russell, A. Torralba, K. P. Murphy, and W. T. Freeman, "LabelMe: A Database and Web-Based Tool for Image Annotation," International Journal of Computer Vision, vol. 77, no. 1–3, pp. 157–173, May 2008. DOI: https://doi.org/10.1007/s11263-007-0090-8
N. Fiedler, M. Bestmann, and N. Hendrich, "ImageTagger: An Open Source Online Platform for Collaborative Image Labeling," in RoboCup 2018: Robot World Cup XXII, vol. 11374, D. Holz, K. Genter, M. Saad, and O. Von Stryk, Eds. Springer International Publishing, 2019, pp. 162–169. DOI: https://doi.org/10.1007/978-3-030-27544-0_13
"DICOM Web Viewer." https://ivmartel.github.io/dwv/.
E. Ziegler et al., "Open Health Imaging Foundation Viewer: An Extensible Open-Source Framework for Building Web-Based Imaging Applications to Support Cancer Research," JCO Clinical Cancer Informatics, vol. 4, Apr. 2020. DOI: https://doi.org/10.1200/CCI.19.00131
"A Data Platform for Medical AI that enables building high-quality datasets and algorithms with lean process | VinDr," Mar. 26, 2021. https://vindr.ai/vindr-lab.
A. Diaz-Pinto et al., "MONAI Label: A framework for AI-assisted interactive labeling of 3D medical images," Medical Image Analysis, vol. 95, July 2024, Art. no. 103207. DOI: https://doi.org/10.1016/j.media.2024.103207
C. Marzahl et al., "EXACT: a collaboration toolset for algorithm-aided annotation of images with annotation version control," Scientific Reports, vol. 11, no. 1, Feb. 2021, Art. no. 4343. DOI: https://doi.org/10.1038/s41598-021-83827-4
U. Rubens et al., "Cytomine: Toward an Open and Collaborative Software Platform for Digital Pathology Bridged to Molecular Investigations," PROTEOMICS – Clinical Applications, vol. 13, no. 1, Jan. 2019, Art. no. 1800057. DOI: https://doi.org/10.1002/prca.201800057
R. Miao, R. Toth, Y. Zhou, A. Madabhushi, and A. Janowczyk, "Quick Annotator: an open‐source digital pathology based rapid image annotation tool," The Journal of Pathology: Clinical Research, vol. 7, no. 6, pp. 542–547, Nov. 2021. DOI: https://doi.org/10.1002/cjp2.229
"Leading Image & Video Data Annotation Platform" CVAT, https://www.cvat.ai.
"Open Source Data Labeling," Label Studio. https://labelstud.io/.
J. Jäger, G. Reus, J. Denzler, V. Wolff, and K. Fricke-Neuderth, "LOST: A flexible framework for semi-automatic image annotation." arXiv, 2019.
"imglab - Image processing and optimization on the fly," imglab. https://imglab.io/.
"MedTAG - Medical Annotation Tool for Diagnostic Reports," GitHub. https://github.com/MedTAG.
D. Strick, C. Garcia, and A. Huang, "Reproducing and Improving CheXNet: Deep Learning for Chest X-ray Disease Classification." arXiv, 2025. DOI: https://doi.org/10.2139/ssrn.5362914
X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, "ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, July 2017, pp. 3462–3471. DOI: https://doi.org/10.1109/CVPR.2017.369
"Radiology Lexicon | NCBO BioPortal." https://bioportal.bioontology.org/ontologies/RADLEX.
"Human Phenotype Ontology | NCBO BioPortal." https://bioportal.bioontology.org/ontologies/HP.
"CXR8 | Powered by Box." https://nihcc.app.box.com/v/ChestXray-NIHCC.
"Random Sample of NIH Chest X-ray Dataset." https://www.kaggle.com/datasets/nih-chest-xrays/sample.
Downloads
How to Cite
License
Copyright (c) 2025 Mona Alshlowi, Samar Alkhuraij, Hajar Alharbi
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
