Presentation of diagnostic accuracy metrics based on classification of artificial intelligence software in radiology

The implementation of artificial intelligence in healthcare is a key direction for technology development in Russia, aimed at improving the quality of medical services and increasing diagnostic accuracy. However, the lack of standards for presenting metrics of diagnostic accuracy of artificial intelligence-based software (AI-based software) complicates comparative analysis and selection of the most suitable software for medical organizations. Therefore, developing a detailed classification of AI-based software is an important task for ensuring safety and quality of medical care, as well as determining the interchangeability of AI-based medical devices.

Purpose: This study aims to develop a clinical classification of AI-based software in radiology.

Materials and Methods: To conduct the study, a comprehensive analysis of available information on AI-based software in radiology was conducted using domestic and foreign databases. In the process of analysis, key aspects were identified, including clinical applicability of AI-based software, diagnostic accuracy of medical devices using AI in radiology.

Results: a clinical classification of AI-based software in the field of radiology was developed. In addition, an important observation regarding the representation of diagnostic accuracy metrics of AI-based software was identified. As a result, the proposed classification was extended and supplemented by defining the level of representation of diagnostic accuracy metrics depending on the clinical classification.

Conclusion: based on the conducted research, a clinical classification of AI-based software has been developed, which provides a unified approach to the presentation of data on diagnostic accuracy by developers. This approach improves the transparency and comparability of information about different AI-based software in medical practice, thereby improving the efficiency and safety of AI-based software use in medical practice. The results of this study have the potential to be scaled to other AI applications and can be used to improve the quality regulation system for AI-enabled medical devices.

1. Bazovye rekomendacii k rabote servisov iskusstvennogo intellekta dlja luchevoj diagnostiki: metodicheskie rekomendacii. S. P. Morozov, L. R. Abuladze, A. E. Andrejchenko, et al. Serija «Luchshie praktiki luchevoj i instrumental'noj diagnostiki». M.: GBUZ «NPKC DiT DZM», 2022; 119: 68. (In Russ.)

2. Morozov SP, Zinchenko VV, Khoruzhaya AN, et al. Standardization of artificial intelligence in healthcare: Russia becomes the leader. Medical doctor and information technology. 2021; 2: 12-19. (In Russ.) doi: 1025881/18110193_2021_2_12.

3. Vasilev YuA, Vladzymyrskyy AV, Omelyanskaya OV, et al. Methodology for testing and monitoring artificial intelligence-based software for medical diagnostics. Digital Diagnostics. 2023; 4(3): 252-267. (In Russ.) doi: 10.17816/DD321971.

4. Kukshev VI. Classification of artificial intelligence systems. Digital Society. 2020; 6: 58-67. (In Russ.) doi: 10.33917/es-6.172.2020.58-67.

5. Jungmann F, Müller L, Hahn F, et al. Commercial AI solutions in detecting COVID-19 pneumonia in chest CT: not yet ready for clinical implementation? European Radiology. Springer Berlin Heidelberg, 2022; 32(5): 3152-3160. doi: 10.1007/s00330-021-08409-4.

6. High-rad.com Internet. iCAS. Available at: https://www.high-rad.com. Accessed 21.05.2024.

7. Corelinesoft.com Internet. AVIEW CAC. Available at: https://www.corelinesoft.com. Accessed 21.05.2024.

8. Ai-rad-companion Internet. AI-Rad Companion (Pulmonary) Available at: https://www.siemens-healthineers.com/en-iq/digital-health-solutions/ai-rad-companion. Accessed 21.05.2024.

9. Florkow MC, Nguyen ChH, Sakkers RJB, et al. Magnetic resonance imaging-based bone imaging of the lower limb: Strategies for generating high-resolution synthetic computed tomography. Journal of Orthopaedic Research. 2024; 42(4): 843-854. doi: 10.1002/jor.25707.

10. Prorodinki.ru. Available at: https://prorodinki.ru. Accessed 21.05.2024. (In Russ.)

11. Webiomed Available at: https://webiomed.ru/products/webiomed-dhra/. Accessed 21.05.2024. (In Russ.)

12. Drokin IS, Ericheva EV, Bukhvalov OL, et al. Experience in developing and implementing an oncological formations searching system using artificial intelligence with the example of X ray computed tomography of the lungs. Medical doctor and information technology. 2019; 3: 48-57. (In Russ.)

13. Sciberia Lungs Electronic resource. Available at: https://sciberia.ru/?product=lungs. Accessed 23.05.2024.

14. Programmnyj modul' dlja analiza mammogramm TU 58.29.31-003-21494354-2021. M.: Platforma tret'e mnenie. 2021: 1-40. (In Russ.)

15. Matvienko A.V. O trebovaniyah k vnedreniyu iskusstvennogo intellekta v zdravoohranenie. PO Cel's. Available at: https://portal.egisz.rosminzdrav.ru/files/26.05.2023_Каспий_Матвиенко. Accessed 21.05.2024. (In Russ.)

16. Klassen V. Ftizisbiomed. Medicinskij servis ftizisbiomed dlya avtomatizirovannogo analiza rentgenogramm organov grudnoj kletki/flyuorogramm (iskusstvennyj medicinskij intellekt). Available at: https://files.sk.ru/navigator/company_files/1121134/1640275876_FBMTTM-RU.pdf. Accessed 24.05.2024. (In Russ.)

17. Dice LR. Measures of the Amount of Ecologic Association Between Species. Ecology. 1945; 26(3): 297-302.

18. Erickson BJ, Kitamura F. Magician’s Corner: 9. Performance Metrics for Machine Learning Models. Radiology: Artificial Intelligence. 2021; 3(3): e200126. doi: 10.1148/ryai.2021200126.

19. de Maissin A, Vallée R, Flamant M, et al. Multi-expert annotation of Crohn’s disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network. Endoscopy International Open. 2021; 9(7): e1136-e1144. doi: 10.1055/a-1468-3964.