Living review
The most comprehensive systematic review of all COVID-19 related diagnostic, prognostic and general population prediction models, including accuracy, quality (risk of bias) and applicability assessment. This living review will be frequently updated in BMJ.
Downloads
Key findings of the Living Review

What is a prediction model?

A prediction model combines various characteristics of a patient and his/her health to estimate an individual’s probability of having a disease, or experiencing a certain event in the future. For example, a model may inform a patient and his/her doctor that he/she has a 90% probability of having COVID-19. After the diagnosis is confirmed, another model estimates that he/she has 20% probability of dying due to COVID-19.

A valid and reliable prediction gives the true probability of a diagnosis or outcome, but this probability is inevitably uncertain for an individual patient. Importantly, however, a valid and reliable prediction supports doctors to make the best possible decisions for an individual patient. Predictions can be used in shared decision making between patients, their families, and doctors. This helps to tailor scarce healthcare resources to those who need them most.

What is the aim of the Living Review?

This Living Review provides an overview of available prediction models for COVID-19, and we performed a quality control of all models. We systematically checked potential issues with (1) the quality of the data used to build the model, (2) the quality of the predictors that form the input for the prediction, (3) the quality of the predicted outcomes, and (4) the quality of statistical data analysis.

The Living Review identifies the best available models to be used in clinical practice. Furthermore, the Living Review guides researchers to improve the quality of their future prediction model studies.

What are the findings?

We screened over 37 000 studies, where a wide array of models was included, from simple pen-and-paper scoring systems, to artificial intelligence that automatically interprets medical images.

To date, we found:

  • 11 models to predict COVID-19 vulnerability in the general population. These models could tailor the use of preventative resources and diagnostic tests.
  • 118 models to predict a diagnosis of COVID-19. When diagnostic capacity is limited, these models could tailor the use of diagnostic tests in patients showing vague or mild symptoms to those with a high probability of COVID-19. Other models of this type speed up the interpretation of CT scans.
  • 107 models to predict whether patients diagnosed with COVID-19 will have an adverse outcome, such as death. These models could guide doctors to make the best possible decisions for individual patients regarding, for example, intensive care support. This helps optimizing the use of scarce hospital resources.

Prediction models are abundant in medical literature. Many are freely available, for example as web applications. Although the majority of studies claim moderate to excellent predictive performance, the Living Review shows that most studies are likely optimistic and biased (i.e., containing systematic error). All types of models, including those based on advanced artificial intelligence, were prone to bias.

However, the quality of prediction models for COVID-19 appears to improve over time. Indeed, we have recently identified four models with a low risk of bias. These models predict the probability of hospital admission or death due to COVID-19 in the general population. The prediction is based on age, ethnicity, UK post code, body mass index, and a medical history. These high-quality models predict the probability of hospital admission or death due to COVID-19 infection and could thus be used to identify people who should get vaccinated as soon as possible. These predictions could also inform policy makers to recommend interventions based on probability, for example regarding occupational exposure.

What should happen now?

To be certain that models can benefit patients and doctors, it is important that they are extensively tested. Validation studies test the models in new patients and quantify how accurate the predictions are.

A prediction model predicts most accurately in the context where it has been developed. Before a prediction model can be used in clinical practice, we strongly recommend to perform a validation study to verify the predictions in a recent and local dataset.

We took initiative to collect and combine individual patient data from multiple studies across the world to provide such validation data and test prediction models before clinical implementation (IPDMA).

Summary details per model
refoutcomeRisk of bias
Feng, Huang et alcovid-19 pneumoniaHigh
DeCapprio, Gartner, et alhospital admission for covid-19 pneumoniaHigh
DeCapprio, Gartner, et alhospital admission for covid-19 pneumoniaHigh
DeCapprio, Gartner, et alhospital admission for covid-19 pneumoniaHigh
Lopez-Rincon, Tonda et alcovid-19 diagnosisHigh
Meng, Wang et al.covid-19 diagnosisHigh
Song, Xu et al.covid-19 diagnosisHigh
Yu, Shao et al.severe covid-19High
Barstugan, Ozkaya et alcovid-19 diagnosisHigh
Chen, Wu et al.covid-19 pneumoniaHigh
Gozes, Frid-Adar et alcovid-19 diagnosisHigh
Jin, Chen et al.covid-19 diagnosisHigh
Jin, Wang et al.covid-19 pneumoniaHigh
Li, Qin et al.covid-19 diagnosisHigh
Shan, Gao et al.otherHigh
Shi, Xia et al.covid-19 pneumoniaHigh
Wang, Kang et al.covid-19 diagnosisHigh
Xu, Jiang et al.covid-19 diagnosisHigh
Ying (Song), Zheng et al.covid-19 diagnosisHigh
Ying (Song), Zheng et al.covid-19 diagnosisHigh
Zheng, Deng et al.covid-19 diagnosisHigh
Bai, Fang et al.progression to severe covid-19High
Carmelo, Oliveiros et al.mortality (in or out of hospital)High
Lu, Hu et al.mortality (in or out of hospital)High
Qi, Jiang et al.length of hospital stayHigh
Qi, Jiang et al.length of hospital stayHigh
Shi, Yu et al.otherHigh
Xie, Hungerford et al.mortalitiy (in hospital)High
Yan, Zhang et al.mortality (in or out of hospital)High
Yuan, Yin et al.mortality (in or out of hospital)High
Martin, Nateqi, et al.covid-19 diagnosisHigh
Sun, Koh, et al.covid-19 diagnosisHigh
Sun, Koh, et al.covid-19 diagnosisHigh
Sun, Koh, et al.covid-19 diagnosisHigh
Sun, Koh, et al.covid-19 diagnosisHigh
Wang, Weng, et al.covid-19 pneumoniaHigh
Wu, Zhang, et al.covid-19 diagnosisHigh
Zhou, Yang, et al.severe covid-19 pneumoniaHigh
Abbas, Abdelsamea, et al.covid-19 diagnosisHigh
Apostolopoulos, Mpesiana.covid-19 diagnosisHigh
Bukhari, Bukhari et al.covid-19 diagnosisHigh
Chaganti, Balachandran, et al.otherHigh
Chaganti, Balachandran, et al.otherHigh
Chaganti, Balachandran, et al.otherHigh
Chaganti, Balachandran, et al.otherHigh
Chowdhury, Rahman et al.otherHigh
Chowdhury, Rahman et al.otherHigh
Chowdhury, Rahman et al.#N/AHigh
Chowdhury, Rahman et al.#N/AHigh
Fu, Yi et al.covid-19 diagnosisHigh
Gozes, Frid-Adar et al.covid-19 diagnosisHigh
Imran, Posokhova, et al.covid-19 diagnosisHigh
Li, Fang, et al.otherHigh
Li, Zhu et al.otherHigh
Hassanien, Mahdy et al.otherUnclear
Tang, Zhao et al.otherHigh
Zhang, Xie, et al.otherHigh
Zhou, Chen et al.covid-19 diagnosisHigh
Huang, Cai et al.otherHigh
Pourhomayoun, Shakibi et al.mortalitiy (in hospital)High
Sakar, Chakrabartimortality (in or out of hospital)High
Zeng, Li et al.progression to severe covid-19Unclear
Zeng, Li et al.progression to severe covid-19Unclear
Al - Najjar, Al-RousanotherHigh
Al - Najjar, Al-Rousanmortality (in or out of hospital)High
Angelov, Soarescovid-19 diagnosisHigh
Arpan, Surya et alcovid-19 diagnosisHigh
Bai, Wang et al.covid-19 diagnosisHigh
Barda, Riesel et almortality (in or out of hospital)High
Bassi, Attuxcovid-19 diagnosisHigh
Batista, Miraglia et alcovid-19 diagnosisHigh
Benchoufi, Bokobza et alotherHigh
Borghesi, MaroldiotherHigh
Born, Brandlecovid-19 diagnosisHigh
Brinati, Campagner et alcovid-19 diagnosisHigh
Brinati, Campagner et alcovid-19 diagnosisHigh
Carr, Bendayan et alprogression to severe covid-19High
Castiglioni, Ippolito et alcovid-19 diagnosisHigh
Chassagnon, Vakalopoulou, et alsevere covid-19High
Chassagnon, Vakalopoulou, et alcomposite outcomeHigh
Chen, Tang et alcovid-19 diagnosisHigh
Colombi, Bodini et alcomposite outcomeHigh
Colombi, Bodini et alcomposite outcomeHigh
Das, Mishra, et almortalitiy (in hospital)High
Diaz-Quijano, Nunes da Silva et alcovid-19 diagnosisHigh
Gong, Ou et alprogression to severe covid-19High
Guiot, Vaidyanathan et al.covid-19 diagnosisHigh
Guo, Liu et al.progression to severe covid-19High
Hu, Liu et almortalitiy (in hospital)High
Hu, Ruan et alcovid-19 diagnosisHigh
Hu, Yao et almortalitiy (in hospital)High
Hu, Yao et almortalitiy (in hospital)High
Islam, Fleischercovid-19 diagnosisHigh
Ji, Zhang et alprogression to severe covid-19High
Jiang, Coffee et alotherHigh
Jiang, Coffee et alotherHigh
Jiang, Coffee et alotherHigh
Jiang, Coffee et alotherHigh
Jiang, Coffee et alotherHigh
Jiang, Coffee et alotherHigh
Jiang, Hu et alotherHigh
Kana, Kana et al.covid-19 diagnosisHigh
Karim, Döhmen et alcovid-19 diagnosisHigh
Kumar, Arora et al.covid-19 diagnosisHigh
Kumar, Arora et al.covid-19 diagnosisHigh
Kurstjens, van der Horst et al.covid-19 diagnosisHigh
Levy, Richardson et al.mortalitiy (in hospital)High
Levy, Richardson et al.mortalitiy (in hospital)High
Levy, Richardson et al.mortalitiy (in hospital)High
Levy, Richardson et al.mortalitiy (in hospital)High
Li, Zhong et alsevere covid-19High
Liu, Fang et al.mortalitiy (in hospital)High
Lyu, Lui et alsevere covid-19 pneumoniaHigh
Lyu, Lui et alsevere covid-19 pneumoniaHigh
Moutounet-Cartancovid-19 pneumoniaHigh
Ozturk, Talo et al.covid-19 pneumoniaHigh
Rehman, Naz et alcovid-19 diagnosisHigh
Rehman, Naz et alcovid-19 diagnosisHigh
Rehman, Naz et alcovid-19 diagnosisHigh
Rehman, Naz et alcovid-19 diagnosisHigh
Rehman, Naz et alcovid-19 diagnosisHigh
Singh, Kumar et alcovid-19 diagnosisHigh
Singh, Valley et alcomposite outcomeHigh
Soares, Villavicencio et alcovid-19 diagnosisHigh
Tordjmann, Mekki et alcovid-19 diagnosisHigh
Ucar, Korkmazcovid-19 diagnosisHigh
Vaid, Somani et alcomposite outcomeHigh
Vazquez Guillamet, Vazquez Guillamet et al.mortalitiy (in hospital)High
Vazquez Guillamet, Vazquez Guillamet et al.otherHigh
Vazquez Guillamet, Vazquez Guillamet et al.composite outcomeHigh
Wang, Deng et alsevere covid-19High
Wu, Gao, et alcovid-19 diagnosisHigh
Zhang, Shi et almortalitiy (in hospital)High
Zhang, Shi et alcomposite outcomeHigh
Zhang, Shi et alcomposite outcomeHigh
Zhao, Wei et alcovid-19 diagnosisHigh
Zhu, Cai et alsevere covid-19High
Ardakani, Kanafi et alcovid-19 diagnosisHigh
Apostolopoulos, Aznaouridis et alcovid-19 diagnosisHigh
Bar, Lecourtois et alcovid-19 diagnosisHigh
Bello-Chavolla, Bahena-Lopez et almortality (in or out of hospital)High
Bi, Su et alprogression to severe covid-19High
Borghesi, Zigliani et almortalitiy (in hospital)High
Burian, Jungmann et alICU admissionHigh
Cecconi, Piovani et alcomposite outcomeHigh
Cheng, Joshi et alICU admissionHigh
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Choi, Ahn et alprogression to severe covid-19High
Clemency, Varughese et alcovid-19 diagnosisHigh
Clemency, Varughese et alcovid-19 diagnosisHigh
Dong, Zhou et alsevere covid-19High
El Asnaoui, Chawki et alotherHigh
Fu, Li et alprogression to severe covid-19High
Galloway, Norton et alcomposite outcomeHigh
Gezer, Ergan et alcovid-19 diagnosisHigh
Gidari, de Socio et alICU admissionHigh
Gidari, de Socio et alICU admissionHigh
Hong, Wu et allength of hospital stayHigh
Huang, Cheng et alprogression to severe covid-19High
Huang, Cheng et alprogression to severe covid-19High
Huang, Wang et alcovid-19 diagnosisHigh
Jehi, Ji et alcovid-19 diagnosisUnclear
Joshi, pejaver et alcovid-19 diagnosisHigh
Kahn, Shah et al.covid-19 diagnosisHigh
Ko, Chung et alcovid-19 pneumoniaHigh
Li, Yang et almortalitiy (in hospital)High
Li, Zeng et alcovid-19 pneumoniaHigh
Li, Zhang et alsevere covid-19High
Liang, Liang et alcomposite outcomeHigh
Liu, Li et alsevere covid-19High
Liu, Shi et alsevere covid-19High
Liu, Wang et alotherHigh
Liu, Wang et alotherHigh
Liu, Wang et alotherHigh
Liu, Zhang et alcomposite outcomeHigh
Liu, Zhang et alcomposite outcomeHigh
Liu, Zhang et alcomposite outcomeHigh
Lorente-Ros, Ruiz et almortality (in or out of hospital)High
Lorente-Ros, Ruiz et almortality (in or out of hospital)high
Luo, Liu et al.mortalitiy (in hospital)High
Luo, Liu et al.mortalitiy (in hospital)Unclear
Luo , Luo et al.covid-19 pneumoniaHigh
Luo, Ying et al.covid-19 diagnosisHigh
Matos J et al.otherHigh
Mazzaccaro, Giacomazzi et alICU admissionHigh
Mazzaccaro, Giacomazzi et alotherHigh
Mazzaccaro, Giacomazzi et almortalitiy (in hospital)High
Mazzaccaro, Giacomazzi et alICU admissionHigh
Mazzaccaro, Giacomazzi et alotherHigh
Mazzaccaro, Giacomazzi et almortalitiy (in hospital)High
McRae, Simmonsmortalitiy (in hospital)High
Mei, Lee et alcovid-19 diagnosisHigh
Menni, Valdes et al.covid-19 diagnosisHigh
Murphy, Smits et alcovid-19 diagnosisHigh
Obeid, Davis et alcovid-19 diagnosisHigh
Pu, Leader et alcovid-19 diagnosisHigh
Rahimzadeh, Attarcovid-19 pneumoniaHigh
Rajaraman, Antanicovid-19 pneumoniaHigh
Rajaraman, Antanicovid-19 pneumoniaHigh
Roland, Gurrola et al.covid-19 diagnosisHigh
Satici, Demirkol et almortality (in or out of hospital)High
Satici, Demirkol et almortality (in or out of hospital)High
Satici, Demirkol et almortality (in or out of hospital)High
Song, Wang et alcovid-19 diagnosisHigh
Sun, Song et al.severe covid-19High
Toraih, Elshazli et alcomposite outcomeHigh
Tuncer, Dogan et al.covid-19 pneumoniaHigh
Vaid, Kalantar et al.covid-19 pneumoniaHigh
Vultaggio, Vivarelli et al.composite outcomeHigh
Wang, Hou et al.mortalitiy (in hospital)High
Wang, Liu et alcovid-19 diagnosisHigh
Wang, Zha et al.covid-19 diagnosisHigh
Wang, Zha et al.length of hospital stayHigh
Wang, Zuo et almortalitiy (in hospital)High
Wang, Zuo et almortalitiy (in hospital)High
Wu, Du et alprogression to severe covid-19High
Wu, Hui, et alcovid-19 pneumoniaHigh
Yang, Shen, et alsevere covid-19High
Yang, Wang, et alsevere covid-19High
Yu, let, et almortalitiy (in hospital)High
Zhang, Liu et al.covid-19 pneumoniaHigh
Zhang, Liu et al.severe covid-19High
Zhang, Qin et al.progression to severe covid-19High
Zheng, Xu et al.length of hospital stayHigh
Zhou, He et al.severe covid-19High
Zou, Li et almortalitiy (in hospital)High
Knight, Ho et almortalitiy (in hospital)Unclear
Clift, Coupland et alotherLow
Clift, Coupland et alotherLow
Clift, Coupland et alhospital admission for covid-19 pneumoniaLow
Clift, Coupland et alhospital admission for covid-19 pneumoniaLow