If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Kaiser Permanente Department of Research & Evaluation, 100 S. Los Robles Ave., Pasadena, California 91101, United States of America.
Department of Research and Evaluation, Kaiser Permanente Southern California, 100 South Los Robles, Pasadena, CA 91101, United States of AmericaSouthern California Permanente Medical Group, Los Angeles Medical Center, 4867 Sunset Blvd, Los Angeles, CA 90027, United States of AmericaKaiser Permanente Bernard J. Tyson School of Medicine, Department of Health Systems Science, 98 S. Los Robles Ave., Pasadena, CA 91101, United States of America
Department of Research and Evaluation, Kaiser Permanente Southern California, 100 South Los Robles, Pasadena, CA 91101, United States of AmericaKaiser Permanente Bernard J. Tyson School of Medicine, Department of Health Systems Science, 98 S. Los Robles Ave., Pasadena, CA 91101, United States of America
Department of Research and Evaluation, Kaiser Permanente Southern California, 100 South Los Robles, Pasadena, CA 91101, United States of AmericaKaiser Permanente Bernard J. Tyson School of Medicine, Department of Health Systems Science, 98 S. Los Robles Ave., Pasadena, CA 91101, United States of America
Develop and validate a risk score using variables available during an Emergency Department (ED) encounter to predict adverse events among patients with suspected COVID-19.
Methods
A retrospective cohort study of adult visits for suspected COVID-19 between March 1 – April 30, 2020 at 15 EDs in Southern California. The primary outcomes were death or respiratory decompensation within 7-days. We used least absolute shrinkage and selection operator (LASSO) models and logistic regression to derive a risk score. We report metrics for derivation and validation cohorts, and subgroups with pneumonia or COVID-19 diagnoses.
Results
26,600 ED encounters were included and 1079 experienced an adverse event. Five categories (comorbidities, obesity/BMI ≥ 40, vital signs, age and sex) were included in the final score. The area under the curve (AUC) in the derivation cohort was 0.891 (95% CI, 0.880–0.901); similar performance was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916). Sensitivity ranging from 100% (Score 0) to 41.7% (Score of ≥15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15). In the subgroups with pneumonia (n = 3252) the AUCs were 0.780 (derivation, 95% CI 0.759–0.801) and 0.832 (validation, 95% CI 0.794–0.870), while for COVID-19 diagnoses (n = 2059) the AUCs were 0.867 (95% CI 0.843–0.892) and 0.837 (95% CI 0.774–0.899) respectively.
Conclusion
Physicians evaluating ED patients with pneumonia, COVID-19, or symptoms suspicious for COVID-19 can apply the COVAS score to assist with decisions to hospitalize or discharge patients during the SARS CoV-2 pandemic.
] and the rapid spread of the disease, data are limited about which patients are at highest risk of clinical decompensation. Presenting symptoms of fever, cough, dyspnea and fatigue have been consistently reported from initial reports from China, Italy, and now the United States [
]. Initial reports describe risk factors and outcomes for admitted patients, but lack information about outcomes for patients outside of the hospital. Frontline physicians who make key decisions about disposition would benefit from rules or models to help them decide who requires hospitalization and who can be safely discharged to home. This is particularly challenging for clinicians who do not have SARS-CoV-2 testing results available at the time hospitalization decisions are made. Historically, scores for bacterial pneumonia, such as the pneumonia severity index (PSI) or CURB-65 have shown to improve the quality and efficiency of healthcare by objectively informing physicians about patient risks of death within 30-days [
]. Given the unique clinical features of COVID-19, there is a need for a novel risk score using data commonly available at the time of the Emergency Department (ED) encounter, in order to inform clinical decisions regarding the disposition of patients with symptoms suspicious for COVID-19.
Substantial surges of infected patients have saturated limited resources (hospital beds, ventilators, etc.) [
] in certain areas of the United States. As a result, this pandemic has changed the flow and availability of usual routine care, even among areas with low COVID-19 prevalence as a result of preparation for potential surges. Usual acute care is currently hampered by the fear that health care workers have of contracting COVID-19 [
]. These circumstances impact the ability of health professionals to even use some of the diagnostic tests found predictive of adverse COVID-19 outcomes [
]. Many of these labs may have limited use in the ED setting due to resource limitations, test availability and long turn-around-times for many results. If a prediction score could be derived with clinical variables readily available at presentation, it would be far more useful for physicians and health systems to evaluate the millions of patients with acute symptoms suspicious for COVID-19.
] to inform frontline health professionals and health systems about which patients are at greatest risk of serious adverse events (death or respiratory decompensation requiring invasive mechanical ventilation, non-invasive ventilation, high-flow oxygen, or oxygen delivered via face mask) among those presenting with pneumonia or other symptoms suspicious for COVID-19 (fever, cough, dyspnea, fatigue, etc.) during the pandemic. This is especially needed during a time when rapid SARS CoV-2 testing is not available for most ED patients. Therefore, in this study we developed and validated the COVAS risk score using patient characteristics associated with adverse outcomes (comorbidities, obesity/BMI, vital signs, age and sex) [
] to predict a patient's risk of a serious adverse event within 7-days of the ED encounter.
2. Methods
2.1 Setting and participants
We conducted a study of all adult ED encounters at 15 community hospitals between March 1, 2020 to April 30, 2020. These dates cover a period of time when substantial community spread of COVID-19 was prevalent in our region [
] but prior to general availability of point-of-care testing results at the time of the ED visit. Study sites were all part of a Southern California integrated health system providing health care to over 1 million ED patients per year (study sites ranging from ≈25,000 to 95,000 ED visits per year). Among all ED visits, approximately 80% are made by health plan members. The integrated system allows for unique data capture of all clinical information included in the electronic health record (EHR) for in-network visits, and for claims data for utilization outside of the integrated system.
ED patient encounters were included for adult (≥ 18 years) patients who had an ICD-10 diagnosis for pneumonia, suspected or confirmed COVID-19 (COVID-19 disease, suspected COVID-19, Exposure to coronavirus or screening for COVID-19) or a symptom suspicious for COVID-19 including cough, fever, fatigue, dyspnea, vomiting, diarrhea, pharyngitis, acute upper respiratory infection, or influenza-like illness (eTable 1). We excluded patients who were not members of the integrated health plan because we do not have accurate comorbidity information for these patients, nor accurate follow-up for outcome information when discharged from the ED. Only the index encounter, or first eligible ED visit, was included in the study sample used to derive and validate the risk score, while subsequent visits were included as outcomes. From the final study sample, we took a stratified random 80% sample for the derivation cohort and reserved the remaining 20% as the validation cohort.
2.2 Criterion standard for COVID-19
The criterion standard to diagnose COVID-19 is a positive SARS-CoV-2 polymerase chain reaction (PCR) test. Our study EDs, like most community EDs, have limited access to rapid PCR testing, therefore most PCR test results are not available for at least 12–24 h requiring ED physicians to make a clinical diagnosis of COVID-19 prior to making a decision to hospitalize or discharge the patient. Additionally, patients with COVID-19 report a variety of symptoms that manifest from infection with the virus [
], therefore any patient with a COVID-19 diagnosis, pneumonia, or the infectious symptoms attributed to COVID-19 (eTable 1) were included in the derivation and validation of the risk score. The score was then tested among subgroups of patients with an initial ED ICD-10 COVID-19 or pneumonia diagnosis, as well as among patients who had a positive SARS-CoV-2 PCR test within 7-days of the encounter.
2.3 Outcome measurements
Our primary outcome was a composite measure of death or respiratory decompensation defined as receipt of mechanical ventilation, non-invasive ventilation, high-flow oxygen or oxygen delivered via face mask, within 7-days (ascertained from mortality files, inpatient flowsheet records and CPT/ICD-10 codes; eTable 2) of ED presentation. Secondary outcomes included positive test results for SARS CoV-2 within 7-days of encounter, hospitalizations (inpatient or observation status), and return ED visits or hospitalizations after discharge from the index ED visit.
2.4 Exposures and statistical analysis
To develop a parsimonious prediction model using information available at the point of care in the ED, we collected data about demographic characteristics, vital signs and comorbid conditions. For comorbidities, we collected ICD-10 codes from the Elixhauser Index (diabetes, hypertension, cancer, etc.) [
]. We did not include laboratory test results, because most patients with suspected COVID-19 did not have such testing, and the receipt of testing likely signals that the patient had been identified as seriously ill by the treating physician.
We included 42 variables in a least absolute shrinkage selection operator (LASSO) regression model to minimize collinearity and to avoid over-fitting. [
] The LASSO model with the smallest predicted residual sum of squares was selected based on 10-fold cross validation. We next included all variables from the optimal LASSO regression in a logistic regression model [
]. Those variables which remained significant were included in the final risk score. To simplify the clinical score among common variables, we combined the Elixhauser diabetes and hypertension categories (complicated and uncomplicated) for analysis. We used standard National Institutes of Health (NIH) body mass index (BMI) categories [<18.5 (underweight), 18.5–24 (normal), 25–29 (overweight), 30–34 (obese class I), 35–39 (obese class II), and ≥ 40 (obese class III, or extreme obesity) [
Obesity Education Initiative Expert Panel on the Identification E, and Treatment of Obesity in Adults. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults: The Evidence Report.
]. Vital sign thresholds were informed based on histograms stratified by deciles, but primarily were informed based on current clinical recommendations and the clinical judgement among co-authors. Age was categorized by decade, but given small sample sizes among extreme older ages, those ages above the highest significant age group were clustered into the highest significant category. We performed 10-fold cross validation to obtain stable parameter estimates for the variables in the logistic regression. Standardized scores were created for each variable in the logistic regression model by dividing each variable coefficient by the smallest coefficient. For ease of use, variables were clustered into 5 categories including comorbidities, obesity/BMI, vital signs, age and sex (COVAS).
To assess discrimination, the area under the curve (AUC) was calculated, and a priori we determined that a final score with an AUC ≥ 0.80 would have meaningful accuracy to guide disposition decisions among suspected COVID-19 patients in the ED. [
] We calculated sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) at each threshold value for the COVAS score from ≥1 to ≥15. Positive likelihood ratios (PLR) and negative likelihood ratios (NLR) were also computed for each possible score. To assess model calibration, we evaluated plots of the observed and predicted values and calculated the Brier score. The final score was validated in a separate data set representing 20% of the study sample. We report the same performance metrics in the validation cohort as reported in the derivation sample.
Descriptive tables report the percentage of patients with a 7-day adverse event at each risk score. We also performed subgroup analyses and assessed risk score performance among patients with a pneumonia diagnosis, or among those with a COVID-19 confirmed diagnosis at the index ED encounter. Last, we describe the number of patients hospitalized and discharged stratified by the primary outcome.
All analyses were performed using SAS (version 9.3; SAS Institute, Inc., Cary, NC). All tests of statistical significance were 2-sided with α = 0.05. This study was approved by the [Institution name blinded for review] Institutional Review Board.
3. Results
The study sample included 26,600 ED patient encounters (21,280 derivation and 5320 validation cohorts) (eFig. 1). In the derivation cohort, the mean age (SD) was 50.7 (19.6), 9053 (42.5%) were men, and 1811 (8.5%) were severely obese (BMI ≥ 40) (Table 1). The subgroups included 2620 (12.3%) patients with pneumonia and 4465 (21.0%) patients with a COVID-related diagnosis. ED encounters resulting in an adverse event within 7-days represented 863 (4.1%) patients (177 died and 686 with respiratory decompensation) and 4226 (19.9%) were hospitalized. Among the relatively small number who received a COVID-19 test (N = 6178), 1105 (17.9%) had a confirmed COVID-19 diagnosis within 7-days (eTable 3). The validation cohort represented a comparable sample (eTables 3 and 4).
Table 1Patient characteristics of derivation cohort. Adverse events were defined as any need for high-flow mask, high-flow nasal cannula, non-invasive ventilation, invasive mechanical ventilation or death within 7-days.
LASSO regression identified 20 variables that were associated with the primary outcome. These variables were included in a logistic regression model and 15 variables from five categories, were included in the final comorbidity, obesity, vital signs, age and sex (COVAS) score. Electrolyte disorders, cardiac arrhythmias, other neurological disorders, weight loss, congestive heart failure, coagulopathy and diabetes were the seven Elixhauser comorbidities included. Severe obesity (BMI ≥ 40), respiratory rate (20–24 and ≥ 25), oxygen saturation (93–94% and ≤ 92%), systolic blood pressure (≤ 105), fever (temperature ≥ 100.4 F), heart rate (≥ 110), age (50–59 and ≥ 60) and male sex were also included in the score (Table 2, Figure 1). The resulting COVAS score included points ranging from one to seven for each variable, with and an overall possible score from 0 to 34.
Table 2Multivariate logistic regression in derivation cohort
The AUC in the derivation cohort was 0.891 (95% CI, 0.880–0.901) (Fig. 2) with sensitivity ranging from 100% (Score 0) to 41.7% (Score ≥ 15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15) (eTable 5). A COVAS threshold of ≥5 had 95.1% sensitivity, 56.0% specificity, while a score of ≥11 had 67.8% sensitivity with 89.5% specificity. The Brier score in the derivation cohort was 0.032 demonstrating good calibration (eFigure 2). Similar performance of the COVAS score was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916; Brier score = 0.031) (eFigures 3 and 4, eTable 6).
Fig. 2The COVAS score performance for the derivation sample (N = 21,280) among all ED visits with pneumonia or symptoms suspicious for COVID-19 between March 1–April 30, 2020, predicting 7-day death or need for critical respiratory intervention. Subgroups of patients with a COVID-19 diagnosis (N = 4465) and those with pneumonia (N = 2620) are also reported.
In the subgroups of patients who were diagnosed with pneumonia, the COVAS score had an AUC of 0.780 (95% CI 0.759–0.801) in the derivation cohort and 0.832 (95% CI 0.794–0.870) in the validation group. For the ED encounters with a COVID-19 ICD-10 diagnosis, the AUC was 0.856 (95% CI 0.838–0.873) in the derivation sample, and 0.859 (95% CI 0.824–0.893) in the validation group (Figs. 2, eFigure 3). See eTables 6 and 7 for subgroup sensitivities and specificities.
Rates of 7-day adverse events among patients in the derivation cohort were calculated for each COVAS score and ranged from 0% (COVAS score 0) to 35.7% (COVAS score ≥ 15) (Table 3). Patients with a COVAS score ≤ 5 had ≤1.5% risk of an adverse event, which may represent a “low-risk” group, and patients with a score ≥ 12 have a ≥ 15% risk representing a “high-risk” cohort. Similar rates were observed for the validation cohort (eTable 8).
Table 3Adverse events (death or need for critical respiratory intervention) within 7-days of an emergency department visit for symptoms suspicious for COVID-19 from March 1–April 30, 2020, stratified by COVAS score, in the derivation cohort
In this study we derived and validated a risk score (COVAS) to accurately predict death or respiratory decompensation within 7-days among a sample of ED patients with pneumonia or symptoms suspicious for COVID-19. The COVAS score uses variables (comorbidities, obesity/BMI ≥40, vital signs, age and sex) available at the time of acute patient presentation which can inform clinical decisions about which patients may benefit from further diagnostic testing or admission to the hospital. The COVAS score not only performed well in the derivation (AUC 0.891) and validation (AUC 0.895) cohorts, but also in the subgroups of patients with pneumonia (AUC derivation 0.780 and validation 0.832), and most importantly in the subgroup with a COVID-19 related diagnosis at the index ED encounter (AUC derivation 0.856 and validation 0.859).
This study addresses a call for risk scores to guide clinical decisions in the care of patients during the COVID-19 pandemic, using methods that avoid concerns about the quality of previous reports [
]. A strength of our study is the use of a heterogeneous sample which is representative of the types of patients that frontline emergency, urgent care and primary care physicians are currently evaluating, and the 7-day timeline is more relevant than longer periods for acute care decisions. Our study also avoids the current challenges related to the availability of SARS CoV-2 testing, and with the uncertain accuracy of different testing strategies [
], especially at a time when it is challenging to distinguish between COVID-19 and other infections. Targeting an ED patient population adds to the previous reports from hospitalized patient cohorts [
A limitation of this study and the derived COVAS score is the omission of diagnostic laboratory tests which have been associated with COVID-19 severity among hospitalized patients. We acknowledge that including laboratory tests may improve discrimination, but implementation of this approach would require that all patients receive lab tests which may not be necessary nor readily available in a pandemic. Also, the COVAS score may have different results in different patient populations or clinical settings; therefore, future research to validate this score will elucidate its generalizability. However, given the high AUC in our overall sample and subgroups, even a modest reduction in discrimination is likely to be useful. Additionally, the COVAS score does not include social risks that may predict adverse outcomes among at-risk populations, such as those with housing instability or food insecurity [
]. Our study population uses only the initial ED patient encounter for risk assessment, therefore, patients with a return visit may have slightly different risks for an adverse event. Our patient population is also diverse racially and ethnically (45% Hispanic and 13% black) which is a strength of our study, but the COVAS score may have different results in more homogenous populations. Lastly, calculating comorbidities based on ICD codes in the preceding 12 months may pose challenges to physicians calculating the score without electronic assistance. Therefore, we recommend using decision support within an electronic health record to facilitate the accurate calculation of the COVAS score.
The COVAS score discriminates as well, or better, than many scores used in routine clinical practice to inform acute care decisions. Commonly used clinical risk scores like the pneumonia severity index (AUC 0.81), CRB65 (AUC 0.79) and CURB-65 (AUC 0.80), which are routinely used in clinical care [
], are less accurate than the COVAS score for their specific indications. Of note, a more complicated score with many more variables, including a number of lab test results was not as accurate for predicting adverse outcomes among Chinese patients hospitalized with COVID-19 [
Since the derivation of the COVAS score, all 15 EDs included in this study implemented the score into routine practice using an automated decision support calculation within the EPIC based electronic health record. From the time of our reported study, the COVAS score was further validated among ED patients who were tested for COVID-19 during a time in Southern California when the incidence and prevalence increased among the study EDs. This additional validation among our study EDs using the same study population in between May and June 2020 resulted in an AUC of 0.822 (95% CI 0.811–0.833), and with much increased capacity to test it performed well among those tested via SARS CoV-2 PCR (n = 18,379, AUC 0.792, 95% CI 0.780–0.805) and was best among those whose COVID-19 test was positive (n = 2091, AUC 0.841, 95% CI 0.812–0.869).
Though the COVAS score requires future research to validate its performance among different patient populations in different health care settings, we believe it provides useful information to frontline physicians to assist with risk stratification and disposition decisions. A challenge for any risk score is choosing the appropriate threshold (sensitivity and specificity) to guide recommendations for patient care, because both the frequency and consequences of adverse events need to be considered. Using information about adverse events in conjunction with COVAS score sensitivity/specificity tables, physicians and systems can choose different thresholds for categorizing patients into low, moderate and high-risk groups. This may inform disposition decisions, in the same way other risk scores have been applied in the ED. [
] One potential application of the COVAS score may be to objectively identify a moderate risk group of patients who may safely avoid hospitalization, but who may benefit from home O2 monitoring and oxygen supplementation.
In summation, among patients visiting an ED for confirmed or suspicious COVID-19 symptoms, the COVAS score, using information available at the time of patient presentation, may help frontline physicians to identify patients who will experience a serious adverse event within 7-days. Future research is needed to further validate this score in other patient populations and clinical settings.
Author contributions
ALS and MKG conceived the study and obtained research funding. BZH collected and analyzed the data. CN and ST assisted with statistical advice. All included authors assisted with the study design and interpretation of results. ALS drafted the manuscript, and all authors contributed substantially to its revision. ALS takes responsibility of the manuscript as a whole.
Funding source
This research was supported by a grant from the Regional Research Committee of Kaiser Permanente Southern California . Grant No.: KP-RRC-20200401
Obesity Education Initiative Expert Panel on the Identification E, and Treatment of Obesity in Adults. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and Obesity in Adults: The Evidence Report.
30968-2&id=gr1.jpg){kind=link}
30968-2&id=gr2.jpg){kind=link}