[en] BACKGROUND: Since the beginning of the pandemic, hospitals have been constantly overcrowded, with several observed waves of infected cases and hospitalisations. To avoid as much as possible this situation, efficient tools to facilitate the diagnosis of COVID-19 are needed.
OBJECTIVE: To evaluate and compare prediction models to diagnose COVID-19 identified in a systematic review published recently using performance indicators such as discrimination and calibration measures.
METHODS: A total of 1618 adult patients present at two Emergency Department triage centers and for whom qRT-PCR tests had been performed were included in this study. Six previously published models were reconstructed and assessed using diagnostic tests as sensitivity (Se) and negative predictive value (NPV), discrimination (Area Under the Roc Curve (AUROC)) and calibration measures. Agreement was also measured between them using Kappa's coefficient and IntraClass Correlation Coefficient (ICC). A sensitivity analysis has been conducted by waves of patients.
RESULTS: Among the 6 selected models, those based only on symptoms and/or risk exposure were found to be less efficient than those based on biological parameters and/or radiological examination with smallest AUROC values (< 0.80). However, all models showed good calibration and values above > 0.75 for Se and NPV but poor agreement (Kappa and ICC < 0.5) between them. The results of the first wave were similar to those of the second wave.
CONCLUSION: Although quite acceptable and similar results were found between all models, the importance of radiological examination was also emphasized, making it difficult to find an appropriate triage system to classify patients at risk for COVID-19.
Disciplines :
Public health, health care sciences & services
Author, co-author :
Dardenne, Nadia ✱; Université de Liège - ULiège > Santé publique : de la Biostatistique à la Promotion de la Santé ; Université de Liège - ULiège > Département des sciences de la santé publique > Biostatistique
Locquet, Médéa ✱; Université de Liège - ULiège > Unité de recherche Santé publique, épidémiologie et économie de la santé (URSAPES)
Diep, Anh Nguyet ; Université de Liège - ULiège > Santé publique : de la Biostatistique à la Promotion de la Santé
GILBERT, Allison ; Centre Hospitalier Universitaire de Liège - CHU > > Service des urgences
Delrez, Sophie ; Centre Hospitalier Universitaire de Liège - CHU > > Service des urgences
BEAUDART, Charlotte ; Université de Liège - ULiège > Unité de recherche Santé publique, épidémiologie et économie de la santé (URSAPES) ; Université de Liège - ULiège > Département des sciences de la santé publique > Santé publique, Epidémiologie et Economie de la santé
Brabant, Christian ; Université de Liège - ULiège > Département de Psychologie > Méta-recherche et éthique de la méthodologie quantitative
GHUYSEN, Alexandre ; Centre Hospitalier Universitaire de Liège - CHU > > Service des urgences
Donneau, Anne-Françoise ✱; Université de Liège - ULiège > Santé publique : de la Biostatistique à la Promotion de la Santé ; Université de Liège - ULiège > Département des sciences de la santé publique > Biostatistique
BRUYERE, Olivier ✱; Université de Liège - ULiège > Unité de recherche Santé publique, épidémiologie et économie de la santé (URSAPES) ; Université de Liège - ULiège > Département des sciences de la santé publique > Santé publique, Epidémiologie et Economie de la santé
✱ These authors have contributed equally to this work.
Language :
English
Title :
Clinical prediction models for diagnosis of COVID-19 among adult patients: a validation and agreement study.
WHO Coronavirus (COVID-19) Dashboard|WHO Coronavirus (COVID-19) Dashboard With Vaccination Data [Internet]. [cited 2021 Apr 20]. Available from: https://covid19.who.int/.
Coronavirus [Internet]. [cited 2021 Apr 9]. Available from: https://www.who.int/health-topics/coronavirus#tab=tab_3.
Khiabani K, Amirzade-Iranaq MH. Are saliva and deep throat sputum as reliable as common respiratory specimens for SARS-CoV-2 detection? A systematic review and meta-analysis. Am J Infect Control [Internet]. 2021 Mar [cited 2021 Apr 9]; Available from: https://linkinghub.elsevier.com/retrieve/pii/S0196655321001401.
Soedarsono S, Febriani A, Hasan H, Widyoningroem A. Management of severe COVID-19 patient with negative RT-PCR for SARS-CoV-2: role of clinical, radiological, and serological diagnosis. Radiol Case Reports [Internet]. 2021 Mar [cited 2021 Apr 9]; Available from: https://linkinghub.elsevier.com/retrieve/pii/S1930043321001874.
Arevalo-Rodriguez I, Buitrago-Garcia D, Simancas-Racines D, Zambrano-Achig P, Del Campo R, Ciapponi A, et al. False-negative results of initial RT-PCR assays for COVID-19: a systematic review. PLoS ONE. 2020;15(12):e0242958. DOI: 10.1371/journal.pone.0242958
What tests could potentially be used for the screening, diagnosis and monitoring of COVID-19 and what are their advantages and disadvantages? - The Centre for Evidence-Based Medicine [Internet]. [cited 2021 Apr 9]. Available from: https://www.cebm.net/covid-19/what-tests-could-potentially-be-used-for-the-screening-diagnosis-and-monitoring-of-covid-19-and-what-are-their-advantages-and-disadvantages/.
Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. BMJ. 2020;369:26.
Gupta RK, Marks M, Samuels THA, Luintel A, Rampling T, Chowdhury H, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J. 2020;56(6). 10.1183/13993003.03498-2020.
Locquet M, Diep AN, Beaudart C, Dardenne N, Brabant C, Bruyère O, et al. A systematic review of prediction models to diagnose COVID-19 in adults admitted to healthcare centers. Arch Public Health. 2021;79:1–15. 10.1186/s13690-021-00630-3. DOI: 10.1186/s13690-021-00630-3
Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, et al. PROBAST: a tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. 2019;170(1):51. DOI: 10.7326/M18-1376
Vieceli T, de OliveiraFilho CM, Berger M, Saadi MP, Salvador PA, Anizelli LB, et al. A predictive score for COVID-19 diagnosis using clinical, laboratory and chest image data. Brazilian J Infect Dis. 2020;24(4):343–8. DOI: 10.1016/j.bjid.2020.06.009
Tordjman M, Mekki A, Mali RD, Saab I, Chassagnon G, Guillo E, et al. Pre-test probability for SARS-Cov-2-related infection score: the PARIS score. PLoS ONE. 2020;15:e0243342. DOI: 10.1371/journal.pone.0243342
Kurstjens S, van der Horst A, Herpers R, Geerits MWL, Kluiters-De Hingh YCM, Göttgens EL, et al. Rapid identification of SARS-CoV-2-infected patients at the emergency department using routine testing. Clin Chem Lab Med. 2020;58(9):1587–93. DOI: 10.1515/cclm-2020-0593
Aldobyany A, Touman A, Ghaleb N, Alsaggaf R, Murtaza N, Hamada A, et al. Correlation between the COVID-19 respiratory triage score and SARS-COV-2 PCR test. Front Med. 2020;7. Available from: https://pubmed.ncbi.nlm.nih.gov/33365322/.
Nakakubo S, Suzuki M, Kamada K, Yamashita Y, Nakamura J, Horii H, et al. Proposal of COVID-19 clinical risk score for the management of suspected COVID-19 cases: a case control study. BMC Infect Dis. 2020;20(1).
Fink DL, Khan PY, Goldman N, Cai J, Hone L, Mooney C, et al. Development and internal validation of a diagnostic prediction model for COVID-19 at time of admission to hospital. QJM An Int J Med. 2020;114:699–705. DOI: 10.1093/qjmed/hcaa305
Saegerman C, Gilbert A, Donneau A-F, Gangolf M, Diep AN, Meex C, et al. Clinical decision support tool for diagnosis of COVID-19 in hospitals. PLoS ONE. 2021;16(3):e0247773. 10.1371/journal.pone.0247773. DOI: 10.1371/journal.pone.0247773
Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A, Chu DK, et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance. 2020;25(3):2000045. 10.2807/1560-7917.ES.2020.25.3.2000045. DOI: 10.2807/1560-7917.ES.2020.25.3.2000045
Hosmer Jr. DW, Lemeshow S, Rodney XS. Applied Logistic Regression, 3rd Edition | Wiley [Internet]. 2013 [cited 2021 Apr 8]. 528 p. Available from: https://www.wiley.com/en-us/Applied+Logistic+Regression%2C+3rd+Edition-p-9780470582473.
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21:128–38. DOI: 10.1097/EDE.0b013e3181c30fb2
Lindhiem O, Petersen IT, Mentch LK, Youngstrom EA. The importance of calibration in clinical psychology. Assessment. 2020;27:840–54. DOI: 10.1177/1073191117752055
Stevens RJ, Poppe KK. Validation of clinical prediction models: what does the “calibration slope” really measure? J Clin Epidemiol. 2020;1(118):93–9. DOI: 10.1016/j.jclinepi.2019.09.016
McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychol Methods. 1996;1(1):30–46. DOI: 10.1037/1082-989X.1.1.30
Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155. DOI: 10.1016/j.jcm.2016.02.012
Measuring Agreement: Models, Methods, and Applications|Wiley [Internet]. [cited 2021 Apr 20]. Available from: https://www.wiley.com/en-us/Measuring+Agreement%3A+Models%2C+Methods%2C+and+Applications-p-9781118078587.
Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8(1):23. DOI: 10.20982/tqmp.08.1.p023
Aguayo GA, Donneau AF, Vaillant MT, Schritz A, Franco OH, Stranges S, et al. Agreement between 35 published frailty scores in the general population. Am J Epidemiol. 2017;186(4):420–34. DOI: 10.1093/aje/kwx061
Liu Y, De A. Multiple imputation by fully conditional specification for dealing with missing data in a large epidemiologic study. Int J Stat Med Res. 2015;4(3):287–95. DOI: 10.6000/1929-6029.2015.04.03.7
Rubin DB, Wiley J, York N, Brisbane C, Singapore T. Multiple imputation for nonresponse in surveys. In: Rubin DB, editor. Hoboken: Wiley. https://doi.org/10.1002/9780470316696.
Harrell E, Frank M. Package “rms” Title Regression Modeling Strategies [Internet]. 2021 [cited 2021 Apr 8]. Available from: https://github.com/harrelfe/rms.
GitHub-BavoDC/CalibrationCurves: The package with the val.prob.ci.2 code, formerly called the val.prob.ci.2 package. [Internet]. [cited 2021 Apr 8]. Available from: https://github.com/BavoDC/CalibrationCurves/.
GitHub-cran/BlandAltmanLeh: This is a read-only mirror of the CRAN R package repository. BlandAltmanLeh—Plots (Slightly Extended) Bland-Altman Plots [Internet]. [cited 2021 Apr 20]. Available from: https://github.com/cran/BlandAltmanLeh.
Vanbelle S. Comparing dependent kappa coefficients obtained on multilevel data. Biometrical J. 2017;59(5):1016–34. DOI: 10.1002/bimj.201600093
Matthias Gamer A, Matthias Gamer M. Package “irr” Title Various Coefficients of Interrater Reliability and Agreement [Internet]. 2019 [cited 2021 Apr 20]. Available from: https://www.r-project.org.
Package “psych” Title Procedures for Psychological, Psychometric, and Personality Research [Internet]. 2021 [cited 2021 Apr 20]. Available from: https://personality-project.org/r/psych/.
Vieceli T, de Oliveira Filho CM, Berger M, Saadi MP, Salvador PA, Anizelli LB, et al. A predictive score for COVID-19 diagnosis using clinical, laboratory and chest image data. Brazilian J Infect Dis. 2020;24(4):343–8. DOI: 10.1016/j.bjid.2020.06.009
Aldobyany A, Touman A, Ghaleb N, Alsaggaf R, Murtaza N, Hamada A, et al. Correlation between the COVID-19 respiratory triage score and SARS-COV-2 PCR test. Front Med. 2020;7: 605689. DOI: 10.3389/fmed.2020.605689
Tordjman M, Mekki A, Mali RD, Saab I, Chassagnon G, Guillo E, et al. Pre-test probability for SARS-Cov-2-related infection score: the PARIS score. PLoS ONE. 2020;15: e0243342. DOI: 10.1371/journal.pone.0243342
Assel M, Sjoberg DD, Vickers AJ. The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models. Diagnostic Progn Res. 2017;1(1):1–7. 10.1186/s41512-017-0020-3. DOI: 10.1186/s41512-017-0020-3
Soltan AAS, Yang J, Pattanshetty R, Novak A, Yang Y, Rohanian O, et al. Real-world evaluation of rapid and laboratory-free COVID-19 triage for emergency care: external validation and pilot deployment of artificial intelligence driven screening. Lancet Digit Heal. 2022;4(4):e266–78. DOI: 10.1016/S2589-7500(21)00272-7
Gilbert A, Ghuysen A. Triage in the time of COVID-19. Lancet Digit Heal. 2022;4:e210–1. 10.1016/S2589-7500. DOI: 10.1016/S2589-7500
Martha JW, Wibowo A, Pranata R. Prognostic value of elevated lactate dehydrogenase in patients with COVID-19: a systematic review and meta-analysis. Postgrad Med J. 2021;1–6. Available from: http://pmj.bmj.com/.
Contou D, Fraissé M, Pajot O, Tirolien JA, Mentec H, Plantefève G. Comparison between first and second wave among critically ill COVID-19 patients admitted to a French ICU: no prognostic improvement during the second wave? Crit Care. 2021;25(1):1–4. 10.1186/s13054-020-03449-6. DOI: 10.1186/s13054-020-03449-6
Iftimie S, Lopez-Azcona AF, Vallverdu I, Hernandez-Flix S, De Febrer G, Parra S, et al. First and second waves of coronavirus disease-19: a comparative study in hospitalized patients in Reus, Spain. PLoS One. 2021;16. Available from: /pmc/articles/PMC8011765/.
Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–26. 10.1002/sim.6787. DOI: 10.1002/sim.6787
Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating (Ser. Statistics for biology and health). Berlin: Springer; 2019. DOI: 10.1007/978-3-030-16399-0
Bar S, Lecourtois A, Diouf M, Goldberg E, Bourbon C, Arnaud E, et al. The association of lung ultrasound images with COVID-19 infection in an emergency room cohort. Anaesthesia. 2020;75(12):1620–5. DOI: 10.1111/anae.15175
McDonald SA, Medford RJ, Basit MA, Diercks DB, Courtney DM. Derivation with internal validation of a multivariable predictive model to predict COVID-19 test results in emergency department patients. Acad Emerg Med. 2020;28(2):206–14. DOI: 10.1111/acem.14182
Huang D, Wang T, Chen Z, Yang H, Yao R, Liang Z. A novel risk score to predict diagnosis with coronavirus disease 2019 (COVID-19) in suspected patients: a retrospective, multicenter, and observational study. J Med Virol. 2020;92(11):2709–17. DOI: 10.1002/jmv.26143
Sung J, Choudry N, Bachour R. Development and validation of a simple risk score for diagnosing COVID-19 in the emergency room. Epidemiol Infect. 2020;148:1–17. DOI: 10.1017/S0950268820002769
Callejon-Leblic MA, Moreno-Luna R, Del Cuvillo A, Reyes-Tejero IM, Garcia-Villaran MA, Santos-Peña M, et al. Loss of smell and taste can accurately predict COVID-19 infection: a machine-learning approach. J Clin Med. 2021;10(4):570. DOI: 10.3390/jcm10040570
Gupta-Wright A, Macleod CK, Barrett J, Filson SA, Corrah T, Parris V, et al. False-negative RT-PCR for COVID-19 and a diagnostic risk score: a retrospective cohort study among patients admitted to hospital. BMJ Open. 2021;11(2): e047110. DOI: 10.1136/bmjopen-2020-047110
Plante TB, Blau AM, Berg AN, Weinberg AS, Jun IC, Tapson VF, et al. Development and external validation of a machine learning tool to rule out COVID-19 among adults in the emergency department using routine blood tests: a large, multicenter, real-world study. J Med Internet Res. 2020;22(12): e24048. DOI: 10.2196/24048
