Advertisement

Identifying patients with symptoms suspicious for COVID-19 at elevated risk of adverse events: The COVAS score

Published:November 04, 2020DOI:https://doi.org/10.1016/j.ajem.2020.10.068

      Abstract

      Objective

      Develop and validate a risk score using variables available during an Emergency Department (ED) encounter to predict adverse events among patients with suspected COVID-19.

      Methods

      A retrospective cohort study of adult visits for suspected COVID-19 between March 1 – April 30, 2020 at 15 EDs in Southern California. The primary outcomes were death or respiratory decompensation within 7-days. We used least absolute shrinkage and selection operator (LASSO) models and logistic regression to derive a risk score. We report metrics for derivation and validation cohorts, and subgroups with pneumonia or COVID-19 diagnoses.

      Results

      26,600 ED encounters were included and 1079 experienced an adverse event. Five categories (comorbidities, obesity/BMI ≥ 40, vital signs, age and sex) were included in the final score. The area under the curve (AUC) in the derivation cohort was 0.891 (95% CI, 0.880–0.901); similar performance was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916). Sensitivity ranging from 100% (Score 0) to 41.7% (Score of ≥15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15). In the subgroups with pneumonia (n = 3252) the AUCs were 0.780 (derivation, 95% CI 0.759–0.801) and 0.832 (validation, 95% CI 0.794–0.870), while for COVID-19 diagnoses (n = 2059) the AUCs were 0.867 (95% CI 0.843–0.892) and 0.837 (95% CI 0.774–0.899) respectively.

      Conclusion

      Physicians evaluating ED patients with pneumonia, COVID-19, or symptoms suspicious for COVID-19 can apply the COVAS score to assist with decisions to hospitalize or discharge patients during the SARS CoV-2 pandemic.

      Keywords

      1. Introduction

      COVID-19 is a pandemic with worldwide impact which has depleted healthcare resources in many areas [
      • Emanuel E.J.
      • Persad G.
      • Upshur R.
      • Thome B.
      • Parker M.
      • Glickman A.
      • et al.
      Fair allocation of scarce medical resources in the time of Covid-19.
      ,
      • Ji Y.
      • Ma Z.
      • Peppelenbosch M.P.
      • Pan Q.
      Potential association between COVID-19 mortality and health-care resource availability.
      ]. Due to constraints in the availability of testing [
      • Lippi G.
      • Simundic A.M.
      • Plebani M.
      Potential preanalytical and analytical vulnerabilities in the laboratory diagnosis of coronavirus disease 2019 (COVID-19).
      ] and the rapid spread of the disease, data are limited about which patients are at highest risk of clinical decompensation. Presenting symptoms of fever, cough, dyspnea and fatigue have been consistently reported from initial reports from China, Italy, and now the United States [
      • Huang C.
      • Wang Y.
      • Li X.
      • Ren L.
      • Zhao J.
      • Hu Y.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      ,
      • Zhou F.
      • Yu T.
      • Du R.
      • Fan G.
      • Liu Y.
      • Liu Z.
      • et al.
      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
      ,
      • Wang D.
      • Hu B.
      • Hu C.
      • Zhu F.
      • Liu X.
      • Zhang J.
      • et al.
      Clinical characteristics of 138 hospitalized patients with 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China.
      ,
      • Onder G.
      • Rezza G.
      • Brusaferro S.
      Case-fatality rate and characteristics of patients dying in relation to COVID-19 in Italy.
      ,
      • Arentz M.
      • Yim E.
      • Klaff L.
      • Lokhandwala S.
      • Riedo F.X.
      • Chong M.
      • et al.
      Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington State.
      ,
      • Richardson S.
      • Hirsch J.S.
      • Narasimhan M.
      • Crawford J.M.
      • McGinn T.
      • Davidson K.W.
      • et al.
      Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City area.
      ,
      • Myers L.C.
      • Parodi S.M.
      • Escobar G.J.
      • Liu V.X.
      Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
      ]. Initial reports describe risk factors and outcomes for admitted patients, but lack information about outcomes for patients outside of the hospital. Frontline physicians who make key decisions about disposition would benefit from rules or models to help them decide who requires hospitalization and who can be safely discharged to home. This is particularly challenging for clinicians who do not have SARS-CoV-2 testing results available at the time hospitalization decisions are made. Historically, scores for bacterial pneumonia, such as the pneumonia severity index (PSI) or CURB-65 have shown to improve the quality and efficiency of healthcare by objectively informing physicians about patient risks of death within 30-days [
      • Sharp A.L.
      • Jones J.P.
      • Wu I.
      • Huynh D.
      • Kocher K.E.
      • Shah N.R.
      • et al.
      CURB-65 performance among admitted and discharged emergency department patients with community-acquired pneumonia.
      ,
      • Jones B.E.
      • Jones J.
      • Bewick T.
      • Lim W.S.
      • Aronsky D.
      • Brown S.M.
      • et al.
      CURB-65 pneumonia severity assessment adapted for electronic decision support.
      ,
      • Chalmers J.D.
      • Singanayagam A.
      • Akram A.R.
      • Choudhury G.
      • Mandal P.
      • Hill A.T.
      Safety and efficacy of CURB65-guided antibiotic therapy in community-acquired pneumonia.
      ,
      • Chalmers J.D.
      • Singanayagam A.
      • Akram A.R.
      • Mandal P.
      • Short P.M.
      • Choudhury G.
      • et al.
      Severity assessment tools for predicting mortality in hospitalised patients with community-acquired pneumonia. Systematic review and meta-analysis.
      ,
      • Chalmers J.D.
      • Akram A.R.
      • Hill A.T.
      Increasing outpatient treatment of mild community-acquired pneumonia: systematic review and meta-analysis.
      ]. Given the unique clinical features of COVID-19, there is a need for a novel risk score using data commonly available at the time of the Emergency Department (ED) encounter, in order to inform clinical decisions regarding the disposition of patients with symptoms suspicious for COVID-19.
      Substantial surges of infected patients have saturated limited resources (hospital beds, ventilators, etc.) [
      • Ji Y.
      • Ma Z.
      • Peppelenbosch M.P.
      • Pan Q.
      Potential association between COVID-19 mortality and health-care resource availability.
      ] in certain areas of the United States. As a result, this pandemic has changed the flow and availability of usual routine care, even among areas with low COVID-19 prevalence as a result of preparation for potential surges. Usual acute care is currently hampered by the fear that health care workers have of contracting COVID-19 [
      • Zhou M.
      • Tang F.
      • Wang Y.
      • Nie H.
      • Zhang L.
      • You G.
      • et al.
      Knowledge, attitude and practice regarding COVID-19 among health care workers in Henan, China.
      ], limited personal protective equipment [
      • The L.
      COVID-19: protecting health-care workers.
      ], and healthcare work flow changes [
      • Kim E.
      Drawing on Israel’s experience organizing volunteers to operationalize drive-through coronavirus testing centers.
      ,
      • Kwon K.T.
      • Ko J.H.
      • Shin H.
      • Sung M.
      • Kim J.Y.
      Drive-through screening center for COVID-19: a safe and efficient screening system against massive community outbreak.
      ,
      • Choi S.
      • Han C.
      • Lee J.
      • Kim S.I.
      • Kim I.B.
      Innovative screening tests for COVID-19 in South Korea.
      ]. These circumstances impact the ability of health professionals to even use some of the diagnostic tests found predictive of adverse COVID-19 outcomes [
      • Lippi G.
      • Plebani M.
      Laboratory abnormalities in patients with COVID-2019 infection.
      ] in hospitalized patients [
      • Liang W.
      • Liang H.
      • Ou L.
      • Chen B.
      • Chen A.
      • Li C.
      • et al.
      Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
      ]. Many of these labs may have limited use in the ED setting due to resource limitations, test availability and long turn-around-times for many results. If a prediction score could be derived with clinical variables readily available at presentation, it would be far more useful for physicians and health systems to evaluate the millions of patients with acute symptoms suspicious for COVID-19.
      Our study addresses the need [
      • Wynants L.
      • Van Calster B.
      • Bonten M.M.J.
      • Collins G.S.
      • Debray T.P.A.
      • De Vos M.
      • et al.
      Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.
      ] to inform frontline health professionals and health systems about which patients are at greatest risk of serious adverse events (death or respiratory decompensation requiring invasive mechanical ventilation, non-invasive ventilation, high-flow oxygen, or oxygen delivered via face mask) among those presenting with pneumonia or other symptoms suspicious for COVID-19 (fever, cough, dyspnea, fatigue, etc.) during the pandemic. This is especially needed during a time when rapid SARS CoV-2 testing is not available for most ED patients. Therefore, in this study we developed and validated the COVAS risk score using patient characteristics associated with adverse outcomes (comorbidities, obesity/BMI, vital signs, age and sex) [
      • Richardson S.
      • Hirsch J.S.
      • Narasimhan M.
      • Crawford J.M.
      • McGinn T.
      • Davidson K.W.
      • et al.
      Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City area.
      ,
      • Myers L.C.
      • Parodi S.M.
      • Escobar G.J.
      • Liu V.X.
      Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
      ] to predict a patient's risk of a serious adverse event within 7-days of the ED encounter.

      2. Methods

      2.1 Setting and participants

      We conducted a study of all adult ED encounters at 15 community hospitals between March 1, 2020 to April 30, 2020. These dates cover a period of time when substantial community spread of COVID-19 was prevalent in our region [
      • Spellberg B.
      • Haddix M.
      • Lee R.
      • Butler-Wu S.
      • Holtom P.
      • Yee H.
      • et al.
      Community Prevalence of SARS-CoV-2 Among Patients With Influenzalike Illnesses Presenting to a Los Angeles Medical Center in March 2020.
      ] but prior to general availability of point-of-care testing results at the time of the ED visit. Study sites were all part of a Southern California integrated health system providing health care to over 1 million ED patients per year (study sites ranging from ≈25,000 to 95,000 ED visits per year). Among all ED visits, approximately 80% are made by health plan members. The integrated system allows for unique data capture of all clinical information included in the electronic health record (EHR) for in-network visits, and for claims data for utilization outside of the integrated system.
      ED patient encounters were included for adult (≥ 18 years) patients who had an ICD-10 diagnosis for pneumonia, suspected or confirmed COVID-19 (COVID-19 disease, suspected COVID-19, Exposure to coronavirus or screening for COVID-19) or a symptom suspicious for COVID-19 including cough, fever, fatigue, dyspnea, vomiting, diarrhea, pharyngitis, acute upper respiratory infection, or influenza-like illness (eTable 1). We excluded patients who were not members of the integrated health plan because we do not have accurate comorbidity information for these patients, nor accurate follow-up for outcome information when discharged from the ED. Only the index encounter, or first eligible ED visit, was included in the study sample used to derive and validate the risk score, while subsequent visits were included as outcomes. From the final study sample, we took a stratified random 80% sample for the derivation cohort and reserved the remaining 20% as the validation cohort.

      2.2 Criterion standard for COVID-19

      The criterion standard to diagnose COVID-19 is a positive SARS-CoV-2 polymerase chain reaction (PCR) test. Our study EDs, like most community EDs, have limited access to rapid PCR testing, therefore most PCR test results are not available for at least 12–24 h requiring ED physicians to make a clinical diagnosis of COVID-19 prior to making a decision to hospitalize or discharge the patient. Additionally, patients with COVID-19 report a variety of symptoms that manifest from infection with the virus [
      • Huang C.
      • Wang Y.
      • Li X.
      • Ren L.
      • Zhao J.
      • Hu Y.
      • et al.
      Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
      ,
      • Zhou F.
      • Yu T.
      • Du R.
      • Fan G.
      • Liu Y.
      • Liu Z.
      • et al.
      Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
      ,
      • Wang D.
      • Hu B.
      • Hu C.
      • Zhu F.
      • Liu X.
      • Zhang J.
      • et al.
      Clinical characteristics of 138 hospitalized patients with 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China.
      ,
      • Arentz M.
      • Yim E.
      • Klaff L.
      • Lokhandwala S.
      • Riedo F.X.
      • Chong M.
      • et al.
      Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington State.
      ,
      • Myers L.C.
      • Parodi S.M.
      • Escobar G.J.
      • Liu V.X.
      Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
      ], therefore any patient with a COVID-19 diagnosis, pneumonia, or the infectious symptoms attributed to COVID-19 (eTable 1) were included in the derivation and validation of the risk score. The score was then tested among subgroups of patients with an initial ED ICD-10 COVID-19 or pneumonia diagnosis, as well as among patients who had a positive SARS-CoV-2 PCR test within 7-days of the encounter.

      2.3 Outcome measurements

      Our primary outcome was a composite measure of death or respiratory decompensation defined as receipt of mechanical ventilation, non-invasive ventilation, high-flow oxygen or oxygen delivered via face mask, within 7-days (ascertained from mortality files, inpatient flowsheet records and CPT/ICD-10 codes; eTable 2) of ED presentation. Secondary outcomes included positive test results for SARS CoV-2 within 7-days of encounter, hospitalizations (inpatient or observation status), and return ED visits or hospitalizations after discharge from the index ED visit.

      2.4 Exposures and statistical analysis

      To develop a parsimonious prediction model using information available at the point of care in the ED, we collected data about demographic characteristics, vital signs and comorbid conditions. For comorbidities, we collected ICD-10 codes from the Elixhauser Index (diabetes, hypertension, cancer, etc.) [
      • Li B.
      • Evans D.
      • Faris P.
      • Dean S.
      • Quan H.
      Risk adjustment performance of charlson and elixhauser comorbidities in ICD-9 and ICD-10 administrative databases.
      ,
      • Quan H.
      • Sundararajan V.
      • Halfon P.
      • Fong A.
      • Burnand B.
      • Luthi J.C.
      • et al.
      Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.
      ]. We used clinical judgement to identify other candidate variables available at the time of triage for ED visits [
      • Arentz M.
      • Yim E.
      • Klaff L.
      • Lokhandwala S.
      • Riedo F.X.
      • Chong M.
      • et al.
      Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington State.
      ,
      • Richardson S.
      • Hirsch J.S.
      • Narasimhan M.
      • Crawford J.M.
      • McGinn T.
      • Davidson K.W.
      • et al.
      Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City area.
      ,
      • Myers L.C.
      • Parodi S.M.
      • Escobar G.J.
      • Liu V.X.
      Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
      ,
      • Zhou M.
      • Tang F.
      • Wang Y.
      • Nie H.
      • Zhang L.
      • You G.
      • et al.
      Knowledge, attitude and practice regarding COVID-19 among health care workers in Henan, China.
      ,
      • Liang W.
      • Liang H.
      • Ou L.
      • Chen B.
      • Chen A.
      • Li C.
      • et al.
      Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
      ]. We did not include laboratory test results, because most patients with suspected COVID-19 did not have such testing, and the receipt of testing likely signals that the patient had been identified as seriously ill by the treating physician.
      We included 42 variables in a least absolute shrinkage selection operator (LASSO) regression model to minimize collinearity and to avoid over-fitting. [
      • Simon N.
      • Tibshirani R.
      Standardization and the group lasso penalty.
      ,
      • Tibshirani R.
      The lasso method for variable selection in the cox model.
      ] The LASSO model with the smallest predicted residual sum of squares was selected based on 10-fold cross validation. We next included all variables from the optimal LASSO regression in a logistic regression model [
      • Liang W.
      • Liang H.
      • Ou L.
      • Chen B.
      • Chen A.
      • Li C.
      • et al.
      Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
      ]. Those variables which remained significant were included in the final risk score. To simplify the clinical score among common variables, we combined the Elixhauser diabetes and hypertension categories (complicated and uncomplicated) for analysis. We used standard National Institutes of Health (NIH) body mass index (BMI) categories [<18.5 (underweight), 18.5–24 (normal), 25–29 (overweight), 30–34 (obese class I), 35–39 (obese class II), and ≥ 40 (obese class III, or extreme obesity) [
      • NHLBI
      Obesity Education Initiative Expert Panel on the Identification E, and Treatment of Obesity in Adults. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults: The Evidence Report.
      ]. Vital sign thresholds were informed based on histograms stratified by deciles, but primarily were informed based on current clinical recommendations and the clinical judgement among co-authors. Age was categorized by decade, but given small sample sizes among extreme older ages, those ages above the highest significant age group were clustered into the highest significant category. We performed 10-fold cross validation to obtain stable parameter estimates for the variables in the logistic regression. Standardized scores were created for each variable in the logistic regression model by dividing each variable coefficient by the smallest coefficient. For ease of use, variables were clustered into 5 categories including comorbidities, obesity/BMI, vital signs, age and sex (COVAS).
      To assess discrimination, the area under the curve (AUC) was calculated, and a priori we determined that a final score with an AUC ≥ 0.80 would have meaningful accuracy to guide disposition decisions among suspected COVID-19 patients in the ED. [
      • Sharp A.L.
      • Jones J.P.
      • Wu I.
      • Huynh D.
      • Kocher K.E.
      • Shah N.R.
      • et al.
      CURB-65 performance among admitted and discharged emergency department patients with community-acquired pneumonia.
      ,
      • Chalmers J.D.
      • Singanayagam A.
      • Akram A.R.
      • Mandal P.
      • Short P.M.
      • Choudhury G.
      • et al.
      Severity assessment tools for predicting mortality in hospitalised patients with community-acquired pneumonia. Systematic review and meta-analysis.
      ,
      • Backus B.E.
      • Six A.J.
      • Kelder J.C.
      • Mast T.P.
      • van den Akker F.
      • Mast E.G.
      • et al.
      Chest pain in the emergency room: a multicenter validation of the HEART score.
      ,
      • Six A.J.
      • Cullen L.
      • Backus B.E.
      • Greenslade J.
      • Parsonage W.
      • Aldous S.
      • et al.
      The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study.
      ] We calculated sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) at each threshold value for the COVAS score from ≥1 to ≥15. Positive likelihood ratios (PLR) and negative likelihood ratios (NLR) were also computed for each possible score. To assess model calibration, we evaluated plots of the observed and predicted values and calculated the Brier score. The final score was validated in a separate data set representing 20% of the study sample. We report the same performance metrics in the validation cohort as reported in the derivation sample.
      Descriptive tables report the percentage of patients with a 7-day adverse event at each risk score. We also performed subgroup analyses and assessed risk score performance among patients with a pneumonia diagnosis, or among those with a COVID-19 confirmed diagnosis at the index ED encounter. Last, we describe the number of patients hospitalized and discharged stratified by the primary outcome.
      All analyses were performed using SAS (version 9.3; SAS Institute, Inc., Cary, NC). All tests of statistical significance were 2-sided with α = 0.05. This study was approved by the [Institution name blinded for review] Institutional Review Board.

      3. Results

      The study sample included 26,600 ED patient encounters (21,280 derivation and 5320 validation cohorts) (eFig. 1). In the derivation cohort, the mean age (SD) was 50.7 (19.6), 9053 (42.5%) were men, and 1811 (8.5%) were severely obese (BMI ≥ 40) (Table 1). The subgroups included 2620 (12.3%) patients with pneumonia and 4465 (21.0%) patients with a COVID-related diagnosis. ED encounters resulting in an adverse event within 7-days represented 863 (4.1%) patients (177 died and 686 with respiratory decompensation) and 4226 (19.9%) were hospitalized. Among the relatively small number who received a COVID-19 test (N = 6178), 1105 (17.9%) had a confirmed COVID-19 diagnosis within 7-days (eTable 3). The validation cohort represented a comparable sample (eTables 3 and 4).
      Table 1Patient characteristics of derivation cohort. Adverse events were defined as any need for high-flow mask, high-flow nasal cannula, non-invasive ventilation, invasive mechanical ventilation or death within 7-days.
      Total N = 21,280No Adverse Event N = 20,417Adverse Event N = 863
      N%N%N%
      Age
       Mean (SD)50.7 (19.6)50.1 (19.4)65.9 (16.6)
       Median (IQR)50 (34, 66)49 (34, 65)68 (56, 78)
      Age group
       <5010,59549.810,44751.214817.2
       50–59343916.2332116.311813.7
       ≥60724634.1664932.659769.2
      Gender
       Female12,22757.511,86458.136342.1
       Male905342.5855341.950057.9
      Race
       Asian/PI20389.619249.411413.2
       Black283413.3272513.410912.6
       Hispanic963745.3927945.535841.5
       White595328.0568527.826831.1
       Other8183.88043.9141.6
      BMI
       Underweight <18.54652.24262.1394.5
       Normal 18.5–24.9516324.3492224.124127.9
       Overweight 25–29.9626829.5606229.720623.9
       Obese 30–39.9683532.1656632.226931.2
       Severely Obese ≥4018118.517058.410612.3
       Missing7383.57363.620.2
      Smoking
       Never or Passive11,82355.611,42255.940146.5
       Quit474422.3444021.830435.2
       Active10715.010275.0445.1
      Missing364217.1352817.311413.2
      Elixhauser comorbidity score, mean (SD)
      Elixhauser comorbidity index based on comorbidities within the year prior.
      3.1 (3.4)2.9 (3.2)7.2 (3.8)
      Heart rate, continuous
      Vital signs based on presenting measurement.
       Mean (SD)89.9 (18.8)89.6 (18.5)97.8 (23.2)
       Median (IQR)88 (76, 101)88 (76, 101)97 (82, 112)
      Heart rate, categorical
      Vital signs based on presenting measurement.
       <11017,55982.516,95583.060470.0
       110–12018098.516878.312214.1
       ≥12113186.211875.813115.2
       Missing5942.85882.960.7
      Systolic BP, continuous
      Vital signs based on presenting measurement.
       Mean (SD)139.2 (22.9)139.4 (22.6)134.5 (28.6)
       Median (IQR)137 (124, 152)138 (124, 152)132 (116, 150)
      Systolic BP, categorical
      Vital signs based on presenting measurement.
       >11517,54382.416,89782.864674.9
       106–11517668.316668.210011.6
       ≤1059994.78904.410912.6
       Missing9724.69644.780.9
      Diastolic BP, continuous
      Vital signs based on presenting measurement.
       Mean (SD)81.0 (15.3)81.2 (15.2)75.8 (18.0)
       Median (IQR)81 (71, 91)81 (72, 91)75 (64, 87)
      Diastolic BP, categorical
      Vital signs based on presenting measurement.
       ≥7015,97175.115,42775.654463.0
       60–69279213.1261912.817320.1
       <6015457.314076.913816.0
       Missing9724.69644.780.9
      Oxygen saturation, continuous
      Vital signs based on presenting measurement.
       Mean (SD)97.6 (2.9)97.7 (2.3)93.2 (7.7)
       Median (IQR)98 (97, 99)98 (97, 99)95 (91, 98)
      Oxygen saturation
      Vital signs based on presenting measurement.
       ≥9519,05789.618,56290.949557.4
       93–948674.17693.89811.4
       ≤926893.24262.126330.5
       Missing6673.16603.270.8
      Respiratory rate, continuous
      Vital signs based on presenting measurement.
       Mean (SD)18.3 (2.9)18.1 (2.6)21.5 (6.0)
       Median (IQR)18 (16, 20)18 (16, 19)20 (18, 24)
      Respiratory rate, categorical
      Vital signs based on presenting measurement.
       <2015,23471.614,86872.836642.4
       20–24460421.6428021.032437.5
       ≥255502.63871.916318.9
       Missing8924.28824.3101.2
      Fever
      Vital signs based on presenting measurement.
      12836.011375.614616.9
      1 Elixhauser comorbidity index based on comorbidities within the year prior.
      2 Vital signs based on presenting measurement.
      LASSO regression identified 20 variables that were associated with the primary outcome. These variables were included in a logistic regression model and 15 variables from five categories, were included in the final comorbidity, obesity, vital signs, age and sex (COVAS) score. Electrolyte disorders, cardiac arrhythmias, other neurological disorders, weight loss, congestive heart failure, coagulopathy and diabetes were the seven Elixhauser comorbidities included. Severe obesity (BMI ≥ 40), respiratory rate (20–24 and ≥ 25), oxygen saturation (93–94% and ≤ 92%), systolic blood pressure (≤ 105), fever (temperature ≥ 100.4 F), heart rate (≥ 110), age (50–59 and ≥ 60) and male sex were also included in the score (Table 2, Figure 1). The resulting COVAS score included points ranging from one to seven for each variable, with and an overall possible score from 0 to 34.
      Table 2Multivariate logistic regression in derivation cohort
      ParameterOR (95% CI)pScore point value
      Point value obtained by standardizing the log-scale estimates using the smallest significant estimate (heart rate ≥ 121).
      Age group (ref ≤50)
       50–591.67 (1.23–2.26)0.0012
       ≥602.02 (1.57–2.61)<0.00013
      Male1.70 (1.43–2.01)<0.00012
      BMI (ref = Normal)
       Underweight <18.50.96 (0.61–1.49)0.8410
       Overweight 25–29.90.80 (0.63–1.01)0.05690
       Obese 30–39.91.15 (0.92–1.45)0.23190
       Severely Obese ≥401.81 (1.32–2.48)0.00022
      Cardiac Arrhythmia1.59 (1.32–1.92)<0.00012
      Congestive Heart Failure1.45 (1.18–1.77)0.00041
      Coagulopathy1.53 (1.24–1.89)<0.00011
      Diabetes1.37 (1.14–1.64)0.00071
      Fluid and Electrolyte Disorders2.50 (2.07–3.02)<0.00013
      Other Neurological Disorders1.78 (1.44–2.20)<0.00012
      Weight Loss1.79 (1.43–2.24)<0.00012
      Heart rate (ref = 〈110)
       110–1201.44 (1.11–1.86)0.00591
       ≥1211.34 (1.02–1.75)0.03491
      Systolic BP (ref≥ 115)
       106–1151.26 (0.96–1.65)0.09610
       ≤1051.64 (1.25–2.14)0.00032
      Oxygen saturation (ref = ≥95)
       93–941.93 (1.46–2.54)<0.00012
       ≤926.79 (5.43–8.48)<0.00017
      Respiratory rate (ref ≤20)
       20–241.68 (1.39–2.02)<0.00012
       ≥254.08 (3.08–5.41)<0.00015
      Fever2.00 (1.57–2.56)<0.00012
      1 Point value obtained by standardizing the log-scale estimates using the smallest significant estimate (heart rate ≥ 121).
      Fig. 1
      Fig. 1COVAS Score derived among patients presenting to the ED with symptoms suspicious for COVID-19 during the pandemic (March 1–April 30, 2020).
      The AUC in the derivation cohort was 0.891 (95% CI, 0.880–0.901) (Fig. 2) with sensitivity ranging from 100% (Score 0) to 41.7% (Score ≥ 15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15) (eTable 5). A COVAS threshold of ≥5 had 95.1% sensitivity, 56.0% specificity, while a score of ≥11 had 67.8% sensitivity with 89.5% specificity. The Brier score in the derivation cohort was 0.032 demonstrating good calibration (eFigure 2). Similar performance of the COVAS score was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916; Brier score = 0.031) (eFigures 3 and 4, eTable 6).
      Fig. 2
      Fig. 2The COVAS score performance for the derivation sample (N = 21,280) among all ED visits with pneumonia or symptoms suspicious for COVID-19 between March 1–April 30, 2020, predicting 7-day death or need for critical respiratory intervention. Subgroups of patients with a COVID-19 diagnosis (N = 4465) and those with pneumonia (N = 2620) are also reported.
      In the subgroups of patients who were diagnosed with pneumonia, the COVAS score had an AUC of 0.780 (95% CI 0.759–0.801) in the derivation cohort and 0.832 (95% CI 0.794–0.870) in the validation group. For the ED encounters with a COVID-19 ICD-10 diagnosis, the AUC was 0.856 (95% CI 0.838–0.873) in the derivation sample, and 0.859 (95% CI 0.824–0.893) in the validation group (Figs. 2, eFigure 3). See eTables 6 and 7 for subgroup sensitivities and specificities.
      Rates of 7-day adverse events among patients in the derivation cohort were calculated for each COVAS score and ranged from 0% (COVAS score 0) to 35.7% (COVAS score ≥ 15) (Table 3). Patients with a COVAS score ≤ 5 had ≤1.5% risk of an adverse event, which may represent a “low-risk” group, and patients with a score ≥ 12 have a ≥ 15% risk representing a “high-risk” cohort. Similar rates were observed for the validation cohort (eTable 8).
      Table 3Adverse events (death or need for critical respiratory intervention) within 7-days of an emergency department visit for symptoms suspicious for COVID-19 from March 1–April 30, 2020, stratified by COVAS score, in the derivation cohort
      Total derivation cohort (N = 21,280)No adverse event (N = 20,417)Adverse event (N = 863)
      ScoreNCol %Cumulative Col %NRow %NRow %
      0284013.313.32840100.000.0
      14632.215.546199.620.4
      2405219.034.6404399.890.2
      317768.342.9176399.3130.7
      4234211.053.9232499.2180.8
      519699.363.2194098.5291.5
      614266.769.9139297.6342.4
      711875.675.4115597.3322.7
      810214.880.298396.3383.7
      97853.783.973793.9486.1
      106893.287.263492.0558.0
      115822.789.953291.4508.6
      124602.292.139084.87015.2
      133741.893.831885.05615.0
      143061.495.325784.04916.0
      15+10084.7100.064864.336035.7

      4. Discussion

      In this study we derived and validated a risk score (COVAS) to accurately predict death or respiratory decompensation within 7-days among a sample of ED patients with pneumonia or symptoms suspicious for COVID-19. The COVAS score uses variables (comorbidities, obesity/BMI ≥40, vital signs, age and sex) available at the time of acute patient presentation which can inform clinical decisions about which patients may benefit from further diagnostic testing or admission to the hospital. The COVAS score not only performed well in the derivation (AUC 0.891) and validation (AUC 0.895) cohorts, but also in the subgroups of patients with pneumonia (AUC derivation 0.780 and validation 0.832), and most importantly in the subgroup with a COVID-19 related diagnosis at the index ED encounter (AUC derivation 0.856 and validation 0.859).
      This study addresses a call for risk scores to guide clinical decisions in the care of patients during the COVID-19 pandemic, using methods that avoid concerns about the quality of previous reports [
      • Wynants L.
      • Van Calster B.
      • Bonten M.M.J.
      • Collins G.S.
      • Debray T.P.A.
      • De Vos M.
      • et al.
      Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.
      ]. A strength of our study is the use of a heterogeneous sample which is representative of the types of patients that frontline emergency, urgent care and primary care physicians are currently evaluating, and the 7-day timeline is more relevant than longer periods for acute care decisions. Our study also avoids the current challenges related to the availability of SARS CoV-2 testing, and with the uncertain accuracy of different testing strategies [
      • Tang Y.W.
      • Schmitz J.E.
      • Persing D.H.
      • Stratton C.W.
      The laboratory diagnosis of COVID-19 infection: current issues and challenges.
      ], especially at a time when it is challenging to distinguish between COVID-19 and other infections. Targeting an ED patient population adds to the previous reports from hospitalized patient cohorts [
      • Arentz M.
      • Yim E.
      • Klaff L.
      • Lokhandwala S.
      • Riedo F.X.
      • Chong M.
      • et al.
      Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington State.
      ,
      • Myers L.C.
      • Parodi S.M.
      • Escobar G.J.
      • Liu V.X.
      Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
      ,
      • Liang W.
      • Liang H.
      • Ou L.
      • Chen B.
      • Chen A.
      • Li C.
      • et al.
      Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
      ].
      A limitation of this study and the derived COVAS score is the omission of diagnostic laboratory tests which have been associated with COVID-19 severity among hospitalized patients. We acknowledge that including laboratory tests may improve discrimination, but implementation of this approach would require that all patients receive lab tests which may not be necessary nor readily available in a pandemic. Also, the COVAS score may have different results in different patient populations or clinical settings; therefore, future research to validate this score will elucidate its generalizability. However, given the high AUC in our overall sample and subgroups, even a modest reduction in discrimination is likely to be useful. Additionally, the COVAS score does not include social risks that may predict adverse outcomes among at-risk populations, such as those with housing instability or food insecurity [
      • Rogers A.
      • Hu Y.R.
      • Schickedanz A.
      • Gottlieb L.
      • Sharp A.
      Understanding High-utilizing patients based on social risk profiles: a latent class analysis within an integrated health system.
      ]. Our study population uses only the initial ED patient encounter for risk assessment, therefore, patients with a return visit may have slightly different risks for an adverse event. Our patient population is also diverse racially and ethnically (45% Hispanic and 13% black) which is a strength of our study, but the COVAS score may have different results in more homogenous populations. Lastly, calculating comorbidities based on ICD codes in the preceding 12 months may pose challenges to physicians calculating the score without electronic assistance. Therefore, we recommend using decision support within an electronic health record to facilitate the accurate calculation of the COVAS score.
      The COVAS score discriminates as well, or better, than many scores used in routine clinical practice to inform acute care decisions. Commonly used clinical risk scores like the pneumonia severity index (AUC 0.81), CRB65 (AUC 0.79) and CURB-65 (AUC 0.80), which are routinely used in clinical care [
      • Sharp A.L.
      • Jones J.P.
      • Wu I.
      • Huynh D.
      • Kocher K.E.
      • Shah N.R.
      • et al.
      CURB-65 performance among admitted and discharged emergency department patients with community-acquired pneumonia.
      ,
      • Jones B.E.
      • Jones J.
      • Bewick T.
      • Lim W.S.
      • Aronsky D.
      • Brown S.M.
      • et al.
      CURB-65 pneumonia severity assessment adapted for electronic decision support.
      ,
      • Chalmers J.D.
      • Singanayagam A.
      • Akram A.R.
      • Choudhury G.
      • Mandal P.
      • Hill A.T.
      Safety and efficacy of CURB65-guided antibiotic therapy in community-acquired pneumonia.
      ] to predict 30-day mortality [
      • Chalmers J.D.
      • Akram A.R.
      • Hill A.T.
      Increasing outpatient treatment of mild community-acquired pneumonia: systematic review and meta-analysis.
      ], are less accurate than the COVAS score for their specific indications. Of note, a more complicated score with many more variables, including a number of lab test results was not as accurate for predicting adverse outcomes among Chinese patients hospitalized with COVID-19 [
      • Liang W.
      • Liang H.
      • Ou L.
      • Chen B.
      • Chen A.
      • Li C.
      • et al.
      Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
      ].
      Since the derivation of the COVAS score, all 15 EDs included in this study implemented the score into routine practice using an automated decision support calculation within the EPIC based electronic health record. From the time of our reported study, the COVAS score was further validated among ED patients who were tested for COVID-19 during a time in Southern California when the incidence and prevalence increased among the study EDs. This additional validation among our study EDs using the same study population in between May and June 2020 resulted in an AUC of 0.822 (95% CI 0.811–0.833), and with much increased capacity to test it performed well among those tested via SARS CoV-2 PCR (n = 18,379, AUC 0.792, 95% CI 0.780–0.805) and was best among those whose COVID-19 test was positive (n = 2091, AUC 0.841, 95% CI 0.812–0.869).
      Though the COVAS score requires future research to validate its performance among different patient populations in different health care settings, we believe it provides useful information to frontline physicians to assist with risk stratification and disposition decisions. A challenge for any risk score is choosing the appropriate threshold (sensitivity and specificity) to guide recommendations for patient care, because both the frequency and consequences of adverse events need to be considered. Using information about adverse events in conjunction with COVAS score sensitivity/specificity tables, physicians and systems can choose different thresholds for categorizing patients into low, moderate and high-risk groups. This may inform disposition decisions, in the same way other risk scores have been applied in the ED. [
      • Sharp A.L.
      • Baecker A.S.
      • Shen E.
      • Redberg R.
      • Lee M.S.
      • Ferencik M.
      • et al.
      Effect of a HEART care pathway on chest pain management within an integrated health system.
      ] One potential application of the COVAS score may be to objectively identify a moderate risk group of patients who may safely avoid hospitalization, but who may benefit from home O2 monitoring and oxygen supplementation.
      In summation, among patients visiting an ED for confirmed or suspicious COVID-19 symptoms, the COVAS score, using information available at the time of patient presentation, may help frontline physicians to identify patients who will experience a serious adverse event within 7-days. Future research is needed to further validate this score in other patient populations and clinical settings.

      Author contributions

      ALS and MKG conceived the study and obtained research funding. BZH collected and analyzed the data. CN and ST assisted with statistical advice. All included authors assisted with the study design and interpretation of results. ALS drafted the manuscript, and all authors contributed substantially to its revision. ALS takes responsibility of the manuscript as a whole.

      Funding source

      This research was supported by a grant from the Regional Research Committee of Kaiser Permanente Southern California . Grant No.: KP-RRC-20200401

      Appendix A. Supplementary data

      References

        • Emanuel E.J.
        • Persad G.
        • Upshur R.
        • Thome B.
        • Parker M.
        • Glickman A.
        • et al.
        Fair allocation of scarce medical resources in the time of Covid-19.
        N Engl J Med. 2020; https://doi.org/10.1056/NEJMsb2005114
        • Ji Y.
        • Ma Z.
        • Peppelenbosch M.P.
        • Pan Q.
        Potential association between COVID-19 mortality and health-care resource availability.
        Lancet Glob Health. 2020; 8e480https://doi.org/10.1016/S2214-109X(20)30068-1
        • Lippi G.
        • Simundic A.M.
        • Plebani M.
        Potential preanalytical and analytical vulnerabilities in the laboratory diagnosis of coronavirus disease 2019 (COVID-19).
        Clin Chem Lab Med. 2020; https://doi.org/10.1515/cclm-2020-0285
        • Huang C.
        • Wang Y.
        • Li X.
        • Ren L.
        • Zhao J.
        • Hu Y.
        • et al.
        Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.
        Lancet. 2020; 395: 497-506https://doi.org/10.1016/S0140-6736(20)30183-5
        • Zhou F.
        • Yu T.
        • Du R.
        • Fan G.
        • Liu Y.
        • Liu Z.
        • et al.
        Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study.
        Lancet. 2020; 395: 1054-1062https://doi.org/10.1016/S0140-6736(20)30566-3
        • Wang D.
        • Hu B.
        • Hu C.
        • Zhu F.
        • Liu X.
        • Zhang J.
        • et al.
        Clinical characteristics of 138 hospitalized patients with 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.1585
        • Onder G.
        • Rezza G.
        • Brusaferro S.
        Case-fatality rate and characteristics of patients dying in relation to COVID-19 in Italy.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.4683
        • Arentz M.
        • Yim E.
        • Klaff L.
        • Lokhandwala S.
        • Riedo F.X.
        • Chong M.
        • et al.
        Characteristics and outcomes of 21 critically ill patients with COVID-19 in Washington State.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.4326
        • Richardson S.
        • Hirsch J.S.
        • Narasimhan M.
        • Crawford J.M.
        • McGinn T.
        • Davidson K.W.
        • et al.
        Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City area.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.6775
        • Myers L.C.
        • Parodi S.M.
        • Escobar G.J.
        • Liu V.X.
        Characteristics of hospitalized adults with COVID-19 in an integrated health care system in California.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.7202
        • Sharp A.L.
        • Jones J.P.
        • Wu I.
        • Huynh D.
        • Kocher K.E.
        • Shah N.R.
        • et al.
        CURB-65 performance among admitted and discharged emergency department patients with community-acquired pneumonia.
        Acad Emerg Med. 2016; 23: 400-405https://doi.org/10.1111/acem.12929
        • Jones B.E.
        • Jones J.
        • Bewick T.
        • Lim W.S.
        • Aronsky D.
        • Brown S.M.
        • et al.
        CURB-65 pneumonia severity assessment adapted for electronic decision support.
        Chest. 2011; 140: 156-163https://doi.org/10.1378/chest.10-1296
        • Chalmers J.D.
        • Singanayagam A.
        • Akram A.R.
        • Choudhury G.
        • Mandal P.
        • Hill A.T.
        Safety and efficacy of CURB65-guided antibiotic therapy in community-acquired pneumonia.
        J Antimicrob Chemother. 2011; 66: 416-423https://doi.org/10.1093/jac/dkq426
        • Chalmers J.D.
        • Singanayagam A.
        • Akram A.R.
        • Mandal P.
        • Short P.M.
        • Choudhury G.
        • et al.
        Severity assessment tools for predicting mortality in hospitalised patients with community-acquired pneumonia. Systematic review and meta-analysis.
        Thorax. 2010; 65: 878-883https://doi.org/10.1136/thx.2009.133280
        • Chalmers J.D.
        • Akram A.R.
        • Hill A.T.
        Increasing outpatient treatment of mild community-acquired pneumonia: systematic review and meta-analysis.
        Eur Respir J. 2011; 37: 858-864https://doi.org/10.1183/09031936.00065610
        • Zhou M.
        • Tang F.
        • Wang Y.
        • Nie H.
        • Zhang L.
        • You G.
        • et al.
        Knowledge, attitude and practice regarding COVID-19 among health care workers in Henan, China.
        J Hosp Infect. 2020; https://doi.org/10.1016/j.jhin.2020.04.012
        • The L.
        COVID-19: protecting health-care workers.
        Lancet. 2020; 395: 922https://doi.org/10.1016/S0140-6736(20)30644-9
        • Kim E.
        Drawing on Israel’s experience organizing volunteers to operationalize drive-through coronavirus testing centers.
        Disaster Med Public Health Prep. 2020; : 1-3https://doi.org/10.1017/dmp.2020.104
        • Kwon K.T.
        • Ko J.H.
        • Shin H.
        • Sung M.
        • Kim J.Y.
        Drive-through screening center for COVID-19: a safe and efficient screening system against massive community outbreak.
        J Korean Med Sci. 2020; 35e123https://doi.org/10.3346/jkms.2020.35.e123
        • Choi S.
        • Han C.
        • Lee J.
        • Kim S.I.
        • Kim I.B.
        Innovative screening tests for COVID-19 in South Korea.
        Clin Exp Emerg Med. 2020; https://doi.org/10.15441/ceem.20.032
        • Lippi G.
        • Plebani M.
        Laboratory abnormalities in patients with COVID-2019 infection.
        Clin Chem Lab Med. 2020; https://doi.org/10.1515/cclm-2020-0198
        • Liang W.
        • Liang H.
        • Ou L.
        • Chen B.
        • Chen A.
        • Li C.
        • et al.
        Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19.
        JAMA Intern Med. 2020; https://doi.org/10.1001/jamainternmed.2020.2033
        • Wynants L.
        • Van Calster B.
        • Bonten M.M.J.
        • Collins G.S.
        • Debray T.P.A.
        • De Vos M.
        • et al.
        Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.
        BMJ. 2020; 369: m1328https://doi.org/10.1136/bmj.m1328
        • Spellberg B.
        • Haddix M.
        • Lee R.
        • Butler-Wu S.
        • Holtom P.
        • Yee H.
        • et al.
        Community Prevalence of SARS-CoV-2 Among Patients With Influenzalike Illnesses Presenting to a Los Angeles Medical Center in March 2020.
        JAMA. 2020; https://doi.org/10.1001/jama.2020.4958
        • Li B.
        • Evans D.
        • Faris P.
        • Dean S.
        • Quan H.
        Risk adjustment performance of charlson and elixhauser comorbidities in ICD-9 and ICD-10 administrative databases.
        BMC Health Serv Res. 2008; 8: 12https://doi.org/10.1186/1472-6963-8-12
        • Quan H.
        • Sundararajan V.
        • Halfon P.
        • Fong A.
        • Burnand B.
        • Luthi J.C.
        • et al.
        Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data.
        Med Care. 2005; 43: 1130-1139https://doi.org/10.1097/01.mlr.0000182534.19832.83
        • Simon N.
        • Tibshirani R.
        Standardization and the group lasso penalty.
        Stat Sin. 2012; 22: 983-1001https://doi.org/10.5705/ss.2011.075
        • Tibshirani R.
        The lasso method for variable selection in the cox model.
        Stat Med. 1997; 16: 385-395https://doi.org/10.1002/(sici)1097-0258(19970228)16:4<385::aid-sim380>3.0.co;2-3
        • NHLBI
        Obesity Education Initiative Expert Panel on the Identification E, and Treatment of Obesity in Adults. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults: The Evidence Report.
        National Heart Lung, and Blood Institute, 1998
        • Backus B.E.
        • Six A.J.
        • Kelder J.C.
        • Mast T.P.
        • van den Akker F.
        • Mast E.G.
        • et al.
        Chest pain in the emergency room: a multicenter validation of the HEART score.
        Crit Pathw Cardiol. 2010; 9: 164-169https://doi.org/10.1097/HPC.0b013e3181ec36d8
        • Six A.J.
        • Cullen L.
        • Backus B.E.
        • Greenslade J.
        • Parsonage W.
        • Aldous S.
        • et al.
        The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study.
        Crit Pathw Cardiol. 2013; 12: 121-126https://doi.org/10.1097/HPC.0b013e31828b327e
        • Tang Y.W.
        • Schmitz J.E.
        • Persing D.H.
        • Stratton C.W.
        The laboratory diagnosis of COVID-19 infection: current issues and challenges.
        J Clin Microbiol. 2020; https://doi.org/10.1128/JCM.00512-20
        • Rogers A.
        • Hu Y.R.
        • Schickedanz A.
        • Gottlieb L.
        • Sharp A.
        Understanding High-utilizing patients based on social risk profiles: a latent class analysis within an integrated health system.
        J Gen Intern Med. 2020; https://doi.org/10.1007/s11606-019-05510-9
        • Sharp A.L.
        • Baecker A.S.
        • Shen E.
        • Redberg R.
        • Lee M.S.
        • Ferencik M.
        • et al.
        Effect of a HEART care pathway on chest pain management within an integrated health system.
        Ann Emerg Med. 2019; 74: 171-180https://doi.org/10.1016/j.annemergmed.2019.01.007