Advertisement

Development and validation of a knowledge-driven risk calculator for critical illness in COVID-19 patients

Published:September 22, 2020DOI:https://doi.org/10.1016/j.ajem.2020.09.051

      Abstract

      Facing the novel coronavirus disease (COVID-19) pandemic, evidence to inform decision-making at all care levels is essential. Based on the results of a study by Petrilli et al., we have developed a calculator using patient data at admission to predict critical illness (intensive care, mechanical ventilation, hospice care, or death).
      We report a retrospective validation of the calculator on 145 consecutive patients admitted with COVID-19 to a single hospital in Israel. Despite considerable differences between the original and validation study populations, of 18 patients with critical illness, 17 were correctly identified (sensitivity: 94.4%, 95% CI, 72.7%–99.9%; specificity: 81.9%, 95% CI, 74.1%–88.2%). Of 127 patients with non-critical illness, 104 were correctly identified.
      Our results indicate that published knowledge can be reliably applied to assess patient risk, potentially reducing the cognitive burden on physicians, and helping policymakers better prepare for future needs.

      1. Introduction

      Facing the rapidly spreading novel coronavirus disease (COVID-19), evidence to inform decision-making at both the clinical and policy-making level is highly needed [
      • Lipsitch M.
      • Swerdlow D.L.
      • Finelli L.
      Defining the epidemiology of covid-19 - studies needed.
      ]. In an impressive work, Petrilli and coworkers [
      • Petrilli C.M.
      • Jones S.A.
      • Yang J.
      • Rajagopalan H.
      • O’Donnell L.
      • Chernyak Y.
      • et al.
      Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study.
      ] have recently reported a multivariable analysis of data collected on 2729 hospitalized patients with COVID-19 at an academic health system in New York City (NY), to predict critical illness (defined as a composite of care in the intensive care unit, use of mechanical ventilation, discharge to hospice, or death). Based on this analysis, we have developed a computed calculator for risk stratification of hospitalized COVID-19 patients. Since rates of severe disease and mortality vary widely by country [], we aimed to validate the risk calculator on a population of COVID-19 patients in Israel (IL).

      2. Methods

      2.1 Risk calculator development

      We used the odds ratios (OR's) obtained by Petrilli et al. through multivariable regression to develop a risk calculator. This was done by directly applying OR's of predictors as observed in the NY population to a multivariate linear model. Predictors are listed on Table 1, Table 2. These included demographics, past medical history, temperature and oxygen saturation on presentation,and selected labs (the first result of c-reactive protein, d-dimer, ferritin, procalcitonin, and troponin). In their analysis, Petrilli et al. found that the risk of developing critical illness was considerably lower for patients admitted later during the study period. This observation was limited to the 6 weeks of the study period and may be the result of unique and local circumstances. Thus, we did not include the week of presentation as a feature in the risk calculator. Petrilli et al. provided OR's for missing data on predictors, and these were used in our risk calculator.
      Table 1Characteristics of hospitalized patients.
      CharacteristicNY populationIL populationp-Value
      Hospitalized, N = 2729Hospitalized, N = 145
      N (%) or median (IQR)N (%) or median (IQR)
      Age63 (51–74)62(46, 71)<0.010
      Male sex1672 (61.3)90(62.1)0.849
      White race1089 (39.9)145(100)<0.010
      Current smoker141 (5.2)18(12.4)<0.010
      Obesity1081 (39.6)33(22.7)<0.010
      Any chronic condition
      Not used in the model.
      2176 (79.7)113(77.9)0.609
      Hyperlipidemia1157 (42.4)60(41.3)0.859
      Hypertension1693 (62.0)59(40.7)<0.010
      Coronary artery disease602 (22.1)14(9.6)<0.010
      Heart failure349 (12.8)1(0)<0.010
      Diabetes950 (34.7)40(27.6)0.058
      Asthma or chronic obstructive pulmonary disorder453 (16.5)14(9.6)<0.010
      Chronic kidney disease580 (21.3)7(4.8)<0.010
      Cancer292 (10.7)16(11.0)0.90
      Temperature at presentation, degrees Celsius37.4 (36.9–38.2)38.5(38.1, 39.0)<0.010
      Temperature ≥ 38 at presentation846 (31.0)86(59.3)<0.010
      Oxygen saturation < 88%422 (15.5)6(4.1)<0.010
      Oxygen saturation at presentation94 (90–96)97(95, 99)
      Ambient air.
      <0.010
      low asterisk Not used in the model.
      low asterisklow asterisk Ambient air.
      Table 2Values of first laboratory tests used in the risk calculator.
      Procalcitonin testing is not performed at HaSharon hospital.
      TestUnitsMedian(IQR)
      Lymphocytes10^3/uL1.0 (0.8, 1.5)
      Creatininemg/dL0.88 (0.75, 1.05)
      C-reactive proteinmg/L0.37 (0.09, 1.10)
      D-dimerng/mL602.0 (336, 1125.7)
      Ferritinng/mL388.1 (161.7, 689.5)
      Troponin-Tng/mL8.0 (6.0, 14.0)
      low asterisk Procalcitonin testing is not performed at HaSharon hospital.

      2.2 Validation process

      We studied patients admitted to the Rabin Medical Center, HaSharon Campus, a teaching medical center at Petach Tikva, Israel. HaSharon Hospital was designated a coronavirus care center, to which COVID-19 patients residing in Central Israel were referred. Included were patients hospitalized starting from March 9, 2020 with confirmed Covid-19, defined as a positive result on real-time reverse transcriptase-polymerase-chain-reaction (RT-PCR) assay of nasopharyngeal or oropharyngeal swab specimens. This was done using a kit by Seegene (Songpa-gu, South Korea). At the time of data extraction, there were no patients still hospitalized who had not reached the composite outcome. Patients' electronic health records were reviewed by two clinicians (MTK and RM) and relevant demographic data, clinical findings and results of the first laboratory and imaging studies done during the admission were extracted. Based on their state at the time of data extraction, patients were determined to have critical illness (at least one of ICU admission, mechanical ventilation or death; no COVID-19 patients were transferred to hospice care) or non-critical disease. Extracted data was then loaded to the risk calculator and analysis of the relationship between predicted risk and actual outcome was performed. A threshold score for predicting critical illness was empirically selected based on actual patient scores such that optimized discrimination between patients with critical and non-critical illness. The risk calculator tool is freely available on the Web at https://coronavirus.kahun.com/Calculate/7.

      2.3 Statistical analysis

      Analysis was done using R (version 3.6.3) [
      • R Core Team
      R: A Language and Environment for Statistical Computing.
      ]. Differences between the NY and IL populations were assessed using the one sample t-test and one sample Wilcoxon signed rank test. Confidence intervals for the calculator performance measures were computed using the “exact” Clopper-Pearson method.
      The study was approved and informed consent waived by the Rabin Medical Center Institutional Review Board (ref. 0339-20-RMC).

      3. Results

      A total of 145 patients were admitted to HaSharon Hospital between March 9, 2020 and May 13, 2020. At the time of analysis,137 had been discharged, none were still hospitalized, and 8 had deceased. Table 1 shows a comparison of the IL and NY populations baseline characteristics. The median age in the IL and NY populations was similar (62 and 63, respectively), and in both, the majority of patients were males (62.1% vs 61.3%). There were no patients of race other than white in the IL population. Fever and oxygen saturation at presentation were significantly higher, on average, in the IL population. Of note, procalcitonin levels were not available for any of the patients, as this test is not routinely performed at HaSharon hospital. Values of other predictors used in the calculator are shown in Table 2. Results of serum troponin, ferritin and d-dimer from the first 48 h of admission were missing in 31 (21.3%), 30 (20.7%), and 40 (27.6%) of the patients, respectively. Missing data was observed more frequently in patients admitted early in the course of the study period, before protocols for routine testing were adopted, and in patients with milder disease.
      Of the 18 patients with critical illness, 17 were correctly identified by the model when a threshold score of 2.2 points was used (sensitivity: 94.4%, 95% CI, 72.7% to 99.9%; specificity: 81.9%, 95% CI, 74.1% to 88.2%). Of the 127 patients with non-critical illness, 104 were correctly identified. The accuracy was 0.83 (95% CI: 0.76–0.89) and the C-statistic, or the area under the ROC (receiver operator characteristic) curve was 0.943.

      4. Discussion

      We used the results of an analysis performed by Petrilli on a population of COVID-19 in NY to develop a risk calculator for critical illness. We report the application of this calculator to make predictions on the outcome of COVID-19 hospitalized patients in Israel. Of note, predictions were fairly accurate, despite considerable differences in baseline characteristics between the two populations.
      Thousands of papers have been published to date on COVID-19. For clinicians, keeping current on medical literature is challenging in normal times but even more demanding during the COVID-19 pandemic, when their abilities are stretched to the limit. Moreover, given the limited capacity of humans (including physicians) to apply probabilistic reasoning [
      • Dolan J.G.
      • Bordley D.R.
      • Mushlin A.I.
      An evaluation of clinicians’ subjective prior probability estimates.
      ], integration of published evidence probably remains mostly at the intuitive level in the minds of clinicians. This state of affairs calls for equipping clinicians with reliable tools to properly evaluate the abundant empirical knowledge, and properly weigh it against their own patients' data. Many current EHR systems document the information required by the calculator in a machine-readable format, allowing for risk assessment to be done automatically (without active physician involvement). Automatic extraction of information and risk calculation performed in the background as patients present to the hospital can reduce the time and effort required from physicians for risk assessment. It can also assure that policy makers are provided a complete view of predicted disease burden for continuous monitoring.
      Public health measures, such as quarantine and shelter-in-place, are guided by the capacity of the healthcare system, with ICU beds and ventilator availability being the “rate limiting factor”. As severe disease often develops during the second week of illness, there is a reported 12 day average lag between illness onset and ICU admission or the development of acute respiratory distress syndrome (ARDS) [

      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study - Lancet. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30566-3/fulltext.

      ]. In an effort to avoid overwhelming the healthcare system capacity, further lifting of restrictions on social interactions is thus delayed until the effects of policy changes can be measured. The turnaround time for policy-makers to get feedback on policy changes is therefore around two weeks. Our results provide reason to believe that future critical illness can be reliably predicted at the time of admission.Such predictions may be used in triage to make sure that high risk patients remain where medical care is rapidly available, or to select patients for (investigational) interventions. At the institutional and healthcare system level, predicting the future burden of critical illness could improve allocation of resources (e.g., personnel and supplies) to preempt shortages. At the State and National levels, such predictions could shorten the feedback turnaround time, allowing policy makers to effectively flatten the epidemic curve and avoid breakdown of medical care, while minimizing restrictions on the workforce to curb the financial crisis.
      Our work has several limitations. It is based on a single study, which, albeit large in scale, includes patients from a single metropolitan area, presenting during a relatively short period of time, in which the local healthcare system was heavily burdened by scores of severely ill patients. The validation population was relatively small and taken from a single hospital. The paucity of critically ill patients in our cohort may contribute to the high area under the ROC curve. As the threshold score for predicting critical illness was empirically determined from the validation population, results may reflect over-fitting. Finally, our dataset lacked information on some laboratory tests which were included in the original analysis, however this could have weakened our results. Further research will be needed to validate our findings in other patient populations.

      5. Conclusion

      We believe that computer-aided risk assessment is a means to put research-derived knowledge to work in the clinical setting. As new research is published, other calculators could be developed. Based on local circumstances, a calculator using predictors from the most relevant study could be used. Moreover, several calculators could be applied in parallel, and calculator voting used to derive potentially more robust predictions. Integration with EHR systems can facilitate automatic data extraction, which could make the use of such tools more user friendly and practical.

      Declaration of Competing Interest

      AC, MZK, RM and GA are employed by Kahun Medical Ltd. All other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

      References

        • Lipsitch M.
        • Swerdlow D.L.
        • Finelli L.
        Defining the epidemiology of covid-19 - studies needed.
        N Engl J Med. 2020; 382: 1194-1196
        • Petrilli C.M.
        • Jones S.A.
        • Yang J.
        • Rajagopalan H.
        • O’Donnell L.
        • Chernyak Y.
        • et al.
        Factors associated with hospital admission and critical illness among 5279 people with coronavirus disease 2019 in New York City: prospective cohort study.
        BMJ. 2020; 369: m1966
      1. World Health Organization COVID-19 situation reports.
        • R Core Team
        R: A Language and Environment for Statistical Computing.
        R Foundation for Statistical Computing, Vienna, Austria2020 (URL)
        • Dolan J.G.
        • Bordley D.R.
        • Mushlin A.I.
        An evaluation of clinicians’ subjective prior probability estimates.
        Med Decis Making. 1986; 6: 216-223
      2. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study - Lancet. https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30566-3/fulltext.