Experience of application artificial intelligence software on 800 thousand fluorographic studies.

Aim: To evaluate the experience of using software based on artificial intelligence technologies as part of the Moscow experiment on the use of innovative technologies in the field of computer vision for the analysis of medical images.

 Material and methods: A retrospective study was conducted. The work includes the conclusion outputs of 3 AI services on 822 thousand fluorographic studies for the period from 05.01.2022 to 29.12.2022. Pathology was present in 28,341 studies (3.4%). The assessment was carried out using quality metrics of binary classifiers and statistical methods. The metrics were assessed depending on the AI services threshold.

 Results: There was a pronounced imbalance between studies with norm and pathology. High values of imbalance-sensitive metrics and low values of imbalance-insensitive metrics were obtained, which was associated with a high rate of false positive and false negative results. By changing the threshold, it was possible to reduce the number of false negative results. For example, one of the AI services, with a threshold of 0.05, correctly identified 46.8% of studies with the norm, and with no false negative results.

 Conclusions: The number of false negative results for the studied versions of AI services is an obstacle to their autonomous implementation into routine practice, which requires their improvement. By optimizing the service threshold, it is possible to achieve error-free identification of 46.8% of studies with the norm, but due to the closed nature of AI services, this method is limited. Further options for optimizing services require additional study.

1. World Heath Organization. Mortality and global health estimates. Available at: https://www.who.int/data/gho/data/themes/mortality-and-global-health-estimates. Accessed Mar 28. 2023.

2. Sung H, Ferlay J, Siege RL, et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2021; 71(3): 209-249. doi: 10.3322/caac.21660.

3. Federal State Statistics Service. Chislo umershikh po osnovnym klassam prichin smerti Available at: https://rosstat.gov.ru/storage/mediabank/demo24-1_2021.xls. Accessed Mar 28. 2023. (In Russ.)

4. Sostoyanie onkologicheskoi pomoshchi naseleniyu Rossii v 2021 godu. Kaprin AD, Starinsky VV, Shakhzadova AO, editors. M.: MNIOI im. P.A. Gertsena; 2022. (In Russ.)

5. Sinitsyn VE, Tyurin IE, Mitkov VV. Consensus Guidelines of Russian Society of Radiology (RSR) and Russian Association of Specialists in Ultrasound Diagnostics in Medicine (RASUDM) «Role of Imaging (X-ray, CT and US) in Diagnosis of COVID-19 Pneumonia» (version 2). Journal of radiology and nuclear medicine. 2020; 101(2): 72-89. (In Russ.)] doi: 10.20862/0042-4676-2020-101-2-72-89.

6. Colman J, Zamfir G, Sheehan F, et al. Chest radiograph characteristics in COVID-19 infection and their association with survival. Eur J Radiol Open. 2021; 8: 100360. doi: 10.1016/j.ejro.2021.100360.

7. ACR. ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection. Available at: https://www.acr.org/Advocacy-and-Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection. Accessed Mar 28. 2023.

8. Wong HYF, Lam HYS, Fong AH, et al. Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19. Radiology. 2020; 296(2): E72-E78. doi: 10.1148/radiol.2020201160.

9. Morozov SP, Protsenko DN, Smetanina SV, et al. Luchevaya diagnostika koronavirusnoi bolezni (COVID-19): organizatsiya, metodologiya, interpretatsiya rezul’tatov: metodicheskie rekomendatsii. M.: GBUZ «NPKC DiT DZM»; 2021. (In Russ.)

10. Segal B, Rubin DM, Rubin G, Pantanowitz A. Evaluating the Clinical Realism of Synthetic Chest X-Rays Generated Using Progressively Growing GANs. SN Comput Sci. 2021; 2(4): 321. doi: 10.1007/s42979-021-00720-7.

11. Rahman T, Chowdhury MEH, Khandakar A, et al. Transfer Learning with Deep Convolutional Neural Network (CNN) for Pneumonia Detection Using Chest X-ray. Applied Sciences. 2020; 10(9): 3233. doi: 10.3390/app10093233.

12. Gazda M, Plavka J, Gazda J, Drotár P. Self-Supervised Deep Convolutional Neural Network for Chest X-Ray Classification. IEEE Access. 2021; 9: 151972-151982. doi: 10.1109/ACCESS.2021.3125324.

13. Wu JT, Wong KCL, Gur Y, et al. Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents. JAMA Netw Open. 2020; 3(10): e2022779. doi: 10.1001/jamanetworkopen.2020.22779.

14. Arzamasov K, Vasilev Y, Vladzymyrskyy A, et al. An International Non-Inferiority Study for the Benchmarking of AI for Routine Radiology Cases: Chest X-ray, Fluorography and Mammography. Healthcare (Basel). 2023; 11(12): 1684. doi: 10.3390/healthcare11121684.

15. Huang XM, Yang BF, Zheng WL, et al. Cost-effectiveness of artificial intelligence screening for diabetic retinopathy in rural China. BMC Health Serv Res. 2022; 22(1): 260. doi: 10.1186/s12913-022-07655-6.

16. Romanovskov YF, Konovalov VK, Kolmogorov VG. Correspondence consultation of X-ray examinations in the Altai Territory. Digital Diagnostics. 2021; 2(1S): 26-27. (In Russ.)] doi: 10.17816/DD20211s26.

17. Order of the Moscow Department of Health №142 of 19 February 2020. «Ob utverzhdenii Poryadka i usloviy provedeniya eksperimenta na ispol’zovaniyu innovatsionnykh tekhnologiy v oblasti komp’yuternogo zreniya dlya analiza meditsinskikh izobrazheniy i dal’neyshego primeneniya v sisteme zdravookhraneniya goroda Moskvy». Available at: https://mosmed.ai/documents/9/Приказ_департамента_здравоохранения_города_Москвы_от_19.02.202015142.pdf_lFf9_slECRah.pdf. Accessed Mar 28. 2023. (In Russ.)

18. Vasilev YA, Vladzymyrskyy AV, Arzamasov KM, et al. Computer vision in radiology: the first stage of the Moscow experiment: Monograph. 2nd edition. M.: Izdatelskie resheniya; 2023. (In Russ.)

19. Certificate RUS of state registration of a computer program №022617324. Morozov SP, Andreichenko AE, Chetverikov SF, et al. Web-based tool for performing ROC analysis of diagnostic test results: №2022616046/04/19/2022. (In Russ.)

20. Luque A, Carrasco A, Martín F, Heras A. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition. 2019; 91: 216-231. doi: 10.1016/j.patcog.2019.02.023.

21. Mortaz E. Imbalance accuracy metric for model selection in multi-class imbalance classification problems. Knowledge-Based Systems. 2020; 210: 106490. doi: 10.1016/j.knosys.2020.106490.

22. Sokolova M, Lapalme G. A systematic analysis of performance measures for classification tasks. Information Processing & Management. 2009; 45: 427-437. doi: 10.1016/j.ipm.2009.03.002.

23. Chicco D, Warrens MJ, Jurman G. The Matthews Correlation Coefficient (MCC) is More Informative Than Cohen’s Kappa and Brier Score in Binary Classification Assessment. IEEE Access. 2021; 9: 78368-78381. doi: 10.1109/ACCESS.2021.3084050.

24. Liz H, Huertas-Tato J, Sánchez-Montañés M, et al. Deep learning for understanding multilabel imbalanced Chest X-ray datasets. Future Generation Computer Systems. 2023; 144: 291-306. doi: 10.1016/j.future.2023.03.005.

25. Minaee S, Kafieh R, Sonka M, et al. Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning. Med Image Anal. 2020; 65: 101794. doi: 10.1016/j.media.2020.101794.

26. Bharodiya AK, Atul MG. An Improved Segmentation Algorithm For X-Ray Images Based On Adaptive Thresholding Classification. International Journal of Scientific & Technology Research. 2019; 8(9): 1617-1623