Article, Emergency Medicine

Failure of vital sign normalization is more strongly associated than single measures with mortality and outcomes

a b s t r a c t

Background: Modified Early Warning Systems (MEWS) scores offer proxies for morbidity and mortality that are easily acquired, but there are limited data on what changing MEWS scores within the ED indicate. We examined the correlation of changing MEWS scores during resuscitation in the ED and in-hospital morbidity and mortality. Methods: We conducted a retrospective analysis on medical ED patients with simplified MEWS scores (without urine output or mental status) admitted to a single academic tertiary care center over one year. Triage-to-Last delta MEWS score and Triage-to-Max delta MEWS scores were calculated and correlated to in-hospital mortality, ICU admission, length of stay (LOS) and diagnosis of sepsis.

Results: Our analysis included 8322 ED patients with an ICU admission rate of 17% and a mortality rate of 2%. Every point of worsened MEWS after triage was more strongly associated with all-cause mortality (OR 2.41, 95% CI 1.96-2.97) than triage MEWS alone (OR 1.33, 95% CI 1.23-1.44; p b 0.001). Likewise, each point of wors- ened MEWS was associated with increased odds of ICU admission (Triage-to-Last: OR 2.12, 95% CI 1.92-2.33 and Triage-to-Max: OR 1.52, 95% CI 1.45-1.60, respectively). Among patients with suspected infection, similar asso- ciations are found.

Conclusions: Dynamic vital signs in the emergency department, as categorized by delta MEWS, and failure to nor- malize abnormalities, were associated with increased mortality, ICU admission, LOS, and the diagnosis of sepsis. Our results suggest that MEWS scores that do not normalize, from triage onward, are more strongly associated with outcome than any single score.

(C) 2019

Introduction

Abbreviation: AUROC, area under the receiver operating characteristic; CCI, Charlson Comorbidity Index; CI, Confidence interval; ED, Emergency department; EMR, Electronic medical record; GCS, Glasgow Coma Scale; HIPAA, Health Insurance Portability and Accountability Act; ICD, International Classification of Diseases; ICU, Intensive care unit; IQR, Interquartile range; IRB, Institutional review board; LOS, Length of stay; MEWS, Modified Early Warning Systems; QI, Quality improvement; qSOFA, Quick Sepsis Related Organ Failure Assessment; RR, Respiratory rate; SBP, Systolic blood pressure; SD, Standard deviation; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology.

* Corresponding author at: Division of Cardiothoracic Surgery, Department of Surgery, University of Utah Health, United States of America.

E-mail addresses: [email protected], @JoeTonnaMD (J.E. Tonna). @JoeTonnaMD (J.E. Tonna).

Sepsis contributes to significant inpatient mortality and is the most expensive cause of hospitalization in the US, at more than $20 billion dollars and contributing 1.6 million Inpatient hospitalizations annually [1-6]. Patients who are not identified until after hospital admission sub- stantiate an even larger proportion of this burden [7]. These delays in identification are often due to occult presentation of sepsis. Among patients with septic shock requiring vasopressors, many are actually normotensive and have non-specific complaints at emergency depart- ment (ED) triage [8]. Early identification of these patients therefore is an area of active research and inquiry.

Modified Early Warning Systems (MEWS) scores as single measures have been used to identify clinically deteriorating patients and those with increased probability of developing sepsis [9]. MEWS vary

https://doi.org/10.1016/j.ajem.2019.12.024

0735-6757/(C) 2019

between institutions but typically are composite scores of vital signs and mental status. MEWS facilitate recognition of occult deterioration through objective quantification of the magnitude of vital sign perturba- tions [10]. In the inpatient setting and at ED triage, the magnitude of static Vital sign abnormalities, quantified by MEWS scores, have been associated with hospital admission, disposition location and mortality [11-14], but there is a paucity of research on what changing vital signs and changing MEWS in the ED mean prior to clinical deterioration [15]. One study demonstrated that pre-hospital hemodynamic variability was associated clinical deterioration [16]. In a second study, ED Blood pressure variability improved Outcome prediction compared to single measurements, but in the same study, repeated measures of other vital sign interestingly failed to improve Prognostic ability over single measures [17]. Clinical trials of Heart rate variability are ongoing in a va- riety of disease processes and show promise [18], but beyond this, it re- mains unknown if changing MEWS scores within the ED correlate with outcome, or improve outcome prediction compared to single/static

measures.

Herein, we investigated the relationship between changing MEWS scores in the ED and the patient outcomes of sepsis, intensive care unit admission, inpatient mortality and hospital length of stay . In this study, our objective was to determine the relationships between dynamic MEWS scores while in the ED and outcomes among Medical patients admitted to the hospital.

Methods

Our analysis is reported according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Guide- lines [19].

Data sharing

To facilitate research reproducibility, replicability, accuracy and transparency, the datasets generated and/or analyzed during the cur- rent study, and the associated analytic code, will be made available in- definitely [DOI 10.17605/OSF.IO/ESJ9K], following publication, to anyone who wishes to access the data for analysis, on the Open Science Foundation [20] (OSF) repository at [https://osf.io/esj9k/]. Data were de-identified in accordance with Section 164.514 of the Health Insur- ance Portability and Accountability Act (HIPAA).

Data source and study population

Data were obtained from the electronic data warehouse at a sin- gle tertiary academic medical center in a manner which has been previously described [21]. Patients were identified if they were seen in the emergency department from November 2017 to Novem- ber 2018 and were subsequently admitted to the hospital during the index visit. All data included were collected as part of routine clinical care [22]. Comprehensive data capture of clinical and outcome vari- ables is ensured through centralized documentation in the electronic health record [EPIC Systems(R), Verona, WI], ongoing routine quality audits in the emergency department and hospital billing charges. Data was extracted from the data warehouse by a trained data scien- tist blinded to the goals of the analysis. Data was complete for all patients.

We included all patients >=18 years of age at the time of presentation to the ED. In order to focus this preliminary analysis on patients with medical diagnoses, including infectious etiologies of their hemodynamic alterations (such as sepsis), rather than on clearly traumatic, burn or or- thopedic etiologies, we excluded patients admitted to the trauma, or- thopedic or Burn units.

Implementation of automated MEWS tracking and threshold determination

Prior to initiation of our research study, in 2015 the institution im- plemented a hospital wide quality improvement (QI) project wherein a variation of MEWS (excluding urine output and mental status) were automatically calculated by the electronic medical record (EMR) for all patients and displayed on the electronic patient board to facilitate pro- vider recognition of deteriorating patients [22]. For ease of automated EMR calculation, our institution chose this variation of MEWS that was composed entirely of objective vital signs collected in real time as op- posed to the standard MEWS or other scores that include subjective as- sessments, laboratory data, or urine output. For the purposes of this manuscript, MEWS refers to this variation without urine output and mental status.

Institutional MEWS are defined in Fig. 1. MEWS (min: 0, max: 12) quantify the degree of vital sign abnormalities within a patient and in- stitutional estimates of inpatient mortality; scores comprise of temper- ature, respiratory rate, heart rate and systolic blood pressure. Vital sign ranges for scores were based on published literature and institutional expert and multi-disciplinary agreement as part of the QI project [23]. After retrospective analysis of the MEWS vital sign ranges applied to 10,746 inpatient visits, MEWS score characteristics were determined that then allowed estimation of the number of alerts per unit per week, the mortality associated with each MEWS score, and the approx- imate sensitivity and specificity for identifying sepsis according to ICD-9 and ICD-10 Diagnosis codes (Supplemental Figs. 1, 2). This data was then presented to stakeholders throughout the institution and thresh- olds were mutually decided upon that when surpassed would trigger the EMR to send pager-based alerts to providers in order to facilitate recognition and response to septic and decompensating patients.

Study variables and outcomes

Our primary outcome is mortality during hospital admission. Second- ary outcomes include need for ICU admission, diagnosis of sepsis, and hospital length of stay (LOS; in 24 h increments). Our primary predic- tors include Triage-to-Last delta MEWS, Triage-to-Max delta MEWS, and as single measures: triage MEWS, max MEWS, and last MEWS. Co- variates included age, sex, severity of illness (Charlson Comorbidity Index [CCI]), influenza diagnosed [yes/no] during hospital stay, sepsis diagnosed [yes/no] during hospital stay, triage presentation date/time, ED entry date/time, admission unit, ED MEWS scores with date/time (triage, maximum, last), hospital unit admission date/time, hospital dis- charge date/time, disposition location (home, home health care, skilled nursing facility, long term acute care, left against medical advice, hos- pice, death). The diagnosis of sepsis was determined after two separate occurrences of the clinical diagnosis of sepsis within an attending physician’s note.

We additionally did a subgroup analysis for patients in whom there was suspected infection. We defined these patients as those who re- ceived antibiotics while in the ED and repeated the analysis among this group with the same covariates and outcome measures.

Delta MEWS calculations

Triage-to-Last delta MEWS is calculated by subtracting the patient’s last MEWS before leaving the ED from their triage MEWS (first – last). Triage-to-Last delta MEWS ranged from -8 (patient trended from MEWS of 0 [normal] to MEWS of 8 [abnormal], i.e., clinically worsened) to 8 (patient trended from MEWS of 8 [abnormal] to MEWS of 0 [nor- mal], i.e., clinically improved). A Triage-to-Max delta MEWS is calculated by subtracting the patient’s max MEWS while in the ED from their triage MEWS (first – max). Triage-to-Max delta MEWS ranged from -8 (patient trended from MEWS of 0 [normal] to MEWS of 8 [abnormal], i.e., clinically worsened) to 0 (patient first MEWS of 8 [abnormal] was also their max MEWS, i.e., clinically unchanged, or

Fig. 1. Utah Modified Early Warning System.

improved). Scores were then scaled for analysis and incremental changes and direction were compared.

We constructed a theorized causal pathway with directed acyclic graphs using Dagitty [24], which is published on the OSF. Our analysis was exploratory (i.e., postdiction) [25], and the results should be consid- ered hypothesis-generating, requiring reproduction with other data.

Statistical analysis

Our primary goal is to analyze the association of changing vital signs, as categorized by the MEWS score, with outcomes, among medical pa- tients. We first describe baseline clinical and demographic characteris- tics of all patients. Descriptive statistics, including counts and percent for binary variables, means [standard deviation; SD] for continuous var- iables, and medians [interquartile range; IQR] for highly skewed contin- uous variables, were used to assess these characteristics. Categorical characteristics were compared using chi-square test or Fisher’s exact test. Continuous characteristics were compared using independent samples t-test or Wilcoxon-Mann-Whitney test. Primary and secondary analyses were assessed using multivariate regression (logistic for binary [mortality, ICU admission], linear for continuous [hospital LOS]) after adjusting for the covariates age, sex, CCI, time in the ED. For models with a primary predictor of delta MEWS, we additionally adjusted for triage MEWS score. Coefficients, 95% CI’s and p-values were reported from all models. Statistical analyses were conducted in STATA 15.1 (Col- lege Park, TX), significance was assessed at the 0.05 level and all tests were two-tailed. This study was approved by the Institutional Review Board (IRB) under #00096120 on January 13th, 2017.

worsening MEWS after triage (Triage-to-Max delta MEWS; OR 1.43, 95% CI 1.29-1.63). Consistent with previous studies, triage MEWS alone was associated with mortality (OR 1.33, 95% CI 1.23-1.44), as was last MEWS alone (OR 1.62, 95% CI 1.49-1.76). In a sensitivity anal- ysis, removing ED LOS as a covariate resulted in a OR for Triage to Last delta MEWS of 2.50 (95%CI 2.03-3.07).

ICU admission

Each point of worsened MEWS before hospital admission, compared to triage (Triage-to-Last delta MEWS), was associated with increased odds of ICU admission (2.12, 95% CI 1.92-2.33) (Table 2, Fig. 3). Like- wise, each point of worsening MEWS score after triage (Triage-to-Max delta MEWS) was associated with increased odds of ICU admission (OR 1.52, 95% CI 1.45-1.60).

LOS

Among survivors (n = 8155), for every point improvement in triage MEWS score before hospital admission (Triage-to-Last delta MEWS), there was an 18 h decrease in hospital LOS (? coeff 0.78, 95% CI 0.60 to 0.97; Supplemental Fig. 3, Table 2), expressed as increase in LOS with in- creasing scores [(? coeff x 24 h increment)]. As expected, every point in- crease in MEWS score after triage (Triage-to-Max delta MEWS) was likewise associated with a 0.29 increment (~7 h) increase in hospital length of stay (? coeff 0.29; 95% CI 0.20-0.39). This was comparable to the LOS increases associated with increases in static scores alone. Last

Results

We identified 8322 ED patients meeting Inclusion/exclusion criteria. Patient characteristics are outlined in Table 1, but briefly, patients were 56 years old [SD, 17.8] and 49% female. Twenty-four percent were diag-

Table 1

Baseline characteristics.

Gender

n (%)

nosed with sepsis during their admission, the median CCI was 3 (IQR 1,7), 17% were admitted to the ICU, and the in-hospital mortality rate was 2%. One thousand three hundred and sixteen patients received an- tibiotics while in the ED and were included in the subgroup analysis for patients in whom infection was suspected.

Mortality

In adjusted analysis, each point of worsened MEWS before hospital admission (Triage-to-Last delta MEWS), adjusted to triage MEWS, was associated with an increased risk of all-cause mortality (OR 2.41, 95% CI 1.96-2.97) (Table 2, Fig. 2). This was additionally true for any

Male 4276 (51.4)

Female 4046 (48.6)

Age, mean (SD) 56.4 (17.8)

Flu 151 (1.8)

Died 167 (2.0)

Sepsis 2035 (24.4)

Admitted to ICU 1443 (17.3)

Length of stay, days, median (IQR) 2.8 (1.7-4.7)

Charlson Comorbidity Index, median (IQR) 3 (1-7)

Triage MEWS Score, median (IQR) 1(0-2)

Max MEWS Score, median (IQR) 2 (1-3)

Last MEWS Score, median (IQR) 0 (0-1)

Time Triage to Max MEWS, minutes, mean (SD) 84.5 (3.4)

Time Triage to Last MEWS, minutes, mean (SD) 335.4 (5.5)

Table 2

Adjusted probabilities of death, ICU admission, sepsis and hospital length of stay.

Death

ICU admission

Sepsis

LOS among survivors

OR (95% CI) p-value

OR (95% CI) p-value

OR (95% CI) p-value

? coeff (95% CI) p-value

Triage MEWSb Max MEWSb Last MEWSb

Triage-to-Last delta MEWSa Triage-to-Max delta MEWSa

1.33 (1.23-1.44) p b 0.000

1.41 (1.31-1.51) p b 0.000

1.62 (1.49-1.76) p b 0.000

2.41 (1.96-2.97) p b 0.000

1.45 (1.29-1.63) p b 0.000

1.37 (1.33-1.42) p b 0.000

1.47 (1.42-1.52) p b 0.000

1.60 (1.54-1.67) p b 0.000

2.12 (1.92-2.33) p b 0.000

1.52 (1.45-1.60) p b 0.000

1.71 (1.65-1.77) p b 0.000

1.80 (1.74-1.87) p b 0.000

1.75 (1.68-1.83) p b 0.000

1.99 (1.81-2.18) p b 0.000

1.68 (1.61-1.78) p b 0.000

0.34 (0.27-0.41) p b 0.000

0.35 (0.29-0.41) p b 0.000

0.50 (0.42-0.58) p b 0.000

0.78 (0.60-0.97) p b 0.000

0.29 (0.20-0.39) p b 0.000

ICU: intensive care unit. LOS: length of stay.

OR: odds ratio.

CI: confidence interval.

CCI: Charlson comorbidity index.

a Adjusted for age, sex, CCI, triage MEWS score.

b Adjusted for age, sex, CCI.

MEWS score carried the second largest associated increase in likelihood of LOS (? coeff 0.50; 95% CI 0.42-0.58).

Development of sepsis

Every point of worsened MEWS before hospital admission was strongly associated with the development of sepsis (OR 1.99, 95% CI: 1.81-2.18) (Table 2, Fig. 4).

Suspected infection

In a sensitivity analysis among the patients who received antibiotics while in the ED, the associations between the various MEWS scores and outcomes were unchanged (Supplemental Table 3).

Comparison of models

Mortality

When comparing how well the different models and their scores perform at predicting inpatient mortality (Supplemental Table 2), as assessed by the area under the receiver operating characteristic

(AUROC), Triage-to-Last delta MEWS score was superior to Triage MEWS alone (AUROC 0.802 vs 0.767; Fig. 5), to Max MEWS score alone (AUROC 0.802 vs 0.786) and to Triage-to-Max delta MEWS (AUROC 0.802 vs 0.786) but not superior to Last MEWS alone (AUROC 0.802 vs 0.801).

Triage-to-Max delta MEWS was superior to triage MEWS alone (AUROC 0.786 vs 0.767), but comparable to Max MEWS (AUROC 0.786 vs 0.786) and Last MEWS (AUROC 0.786 vs 0.801) alone for the outcome of mortality.

ICU admission

For ICU admission, overall, Triage-to-Last delta MEWS and Triage-to- Max delta MEWS were numerically similar to static Max MEWS scores, but were superior to Triage and Last MEWS scores alone (AUROC [0.731 vs 0.704] and [0.734 vs 0.725]) and (AUROC [0.731 vs 0.704] and [0.734

vs 0.725]) (Supplemental Table 2).

Sepsis

For the diagnosis of sepsis, Triage-to-Last and Triage-to-Max delta MEWS scores were superior to static scores for all comparisons (Supple- mental Table 2).

Fig. 2. Probability of death.

Fig. 3. Probability of ICU admission.

Suspected infection

In a sensitivity analysis among the patients who received antibiotics while in the ED, similar comparisons between the various MEWS were observed (Supplemental Table 3).

Discussion

In the emergency department setting it can be difficult to prognosti- cate a patient’s clinical trajectory both during and after their stay in the

ED. While emergency physicians are trained to identify and correct sig- nificant vital sign abnormalities such as profound hypotension or persis- tent tachycardia, subtle vital sign changes often go unnoticed. Quantification of vital sign abnormalities through clinical decision- making tools such as quick Sepsis-related Organ Failure Assessment and MEWS facilitates recognition of these abnormalities [10]. There is a large body of data demonstrating that the magnitude of these vital sign abnormalities at triage and at hospital admission, quan- tified by these clinical decision tools or early warning scores, is also as- sociated with outcome [12,26]. Vital sign abnormalities collected in the

Fig. 4. Probability of development of sepsis.

Fig. 5. Single vs delta MEWS Scores for the prediction of all-cause mortality–Triage vs Triage-to-Last.

first minutes after ED admission can identify patients at risk of an unfa- vorable outcome [27]. Furthermore, electronic track-and-trigger sys- tems utilizing early warning scores have reduced triage-to-diagnosis and diagnosis-to-antibiotic times in patients with sepsis [28], which has itself been associated with improved mortality [29]. In sum, these tools offer guidance in identifying those who are currently septic or at higher risk for inpatient morbidity and mortality. Unfortunately, all are also single static data points that do not incorporate the patient’s trend in their current clinical course.

Repeated measures of vital signs enable assessment of dynamic he- modynamic changes. These assessments are appealing as they should correlate with efficacy of resuscitation and outcome. Though there are limited studies examining this, clinical trials are ongoing. Among stud- ies of repeated vital signs or scores, a recent large retrospective analysis from 12 hospitals found that repeated measurements of qSOFA im- proved Predictive validity for mortality among hospitalized septic pa- tients compared with a single measurement [30]. Our study is distinct in that MEWS is automatically calculable and does not require the sub- jective assessment of mental status inherent in qSOFA. Among compa- rable studies analyzing objective measurements, the most comparable was a study of febrile or suspected septic emergency department pa- tients (n = 359); it found that only mean blood pressure variability, but not heart rate (HR), respiratory rate (RR), or temperature variability, was superior to static measurements for outcome prediction [17]. A pre- hospital study (n = 2586) among non-trauma patients demonstrated that changes in some vital signs (GCS, RR, SBP, Pulse pressure, Shock Index) between prehospital and in-hospital was associated with an in- creased risk of in-hospital mortality [16]. As it has been demonstrated that commonly used early warning systems have been found to be more accurate than the qSOFA score for predicting death and ICU trans- fer, we believe the objective, vital sign based MEWS offers ideal

characteristics for automated serial assessments in the emergency de- partment [31,32].

By utilizing repeated MEWS scores (delta MEWS), we were able to capture variation in vital sign abnormalities and explore the association with outcome. Within our study of 8322 emergency department pa- tients, we found significantly greater predictive value in the degree of change between serially collected MEWS scores compared to static scores at triage for most outcomes, with an AUROC of Triage-to-Last MEWS for mortality of 0.802. In comparison, the diagnostic perfor- mance of positive qSOFA score for predicting 28-day mortality was low in critically ill septic patients, particularly during the early period after ED presentation. Positive qSOFA for predicting 28-day mortality in- creased from AUROC of 0.58 at arrival to 0.61 within 6 h [33]. Our results should be interpreted cautiously. We did not compare our model to other scores, nor to their covariates. Our results should be interpreted to suggest that failure to normalize MEWS scores, from triage onward, was strongly associated with outcome, and was superior to many static scores during emergency department stay. Each point worsening of MEWS scores–either from initial triage to max (Triage-to-Max delta MEWS) or before hospital admission (Triage-to-Last delta MEWS)– was associated with increased mortality, ICU admission, and diagnosis of sepsis.

In our analysis, delta scores were more strongly associated in all comparisons to single MEWS values alone for all outcomes. For the pre- diction of mortality, ICU admission, and diagnosis of sepsis, models uti- lizing delta MEWS scores were superior to many but not all other models. Specifically, delta scores were variously comparable to Max MEWS score and Last MEWs scores alone. Our findings are important and novel in that they suggest that dynamic changes in hemodynamic status during ED stay, as quantified by MEWS scores, are more strongly associated with these outcomes (survival, ICU admission, sepsis) than

the static condition upon initial presentation. Our findings also suggest that the final condition upon leaving the ED (Last MEWS) is, in this anal- ysis, as predictive of outcome as changes during the ED stay.

Future directions

MEWS are a simple and easy to use bedside tool. Implementation of MEWS to guide Clinical decisions has been inconsistently associated with improved patient outcomes [34]. This sentiment was echoed in an- other recent review that found a lack of high-quality comparative stud- ies [35]. To overcome this gap, prospective multicenter studies are needed that test the durability of these findings in other settings and assess the effect of automated MEWS-based decisions on patient out- comes. Accordingly, we encourage researchers to validate our method- ology and findings within other settings.

Limitations

This study was conducted at a single tertiary-care institution that sees a significant number of medically complex patients and is a referral hospital for a very large geographic area (Utah, Idaho, Wyoming, Mon- tana, Northern Nevada). For these reasons the findings in this study may not be applicable to every ED. This study was limited by not having follow-up information for patients discharged from the ED. If these pa- tients died, or returned to another hospital, we did not capture that. It is known that the rate of death after ED discharge is generally low, but this remains probabilistic in our analysis. While we demonstrated the models utilizing delta MEWS had a significantly larger association with the outcomes than single measures (2.4 vs 1.6), the mortality model’s Predictive ability was only marginally improved (AUROC im- proved 0.03); this is unsurprising considering the AUROC is a measure of model prediction ability, and the models are otherwise identical ex- cept for the primary predictor. A final limitation of the study was the number of covariates. As this was an exploratory study, we analyzed hy- pothesized covariates based on our DAG, though there is likely uncon- trolled confounding present at some level.

Conclusions

In this study of 8322 emergency department medical patients ad- mitted to the hospital, we found that dynamic vital signs while in the emergency department, as categorized by MEWS, and specifically fail- ure to normalize abnormalities, were associated with increased mortal- ity, probability of ICU admission, length of stay, and the diagnosis of sepsis. Changes in MEWS during ED stay were superior to static scores at triage for the predicting mortality, ICU admission and sepsis. The final MEWS score in the ED was strongly associated with outcome, com- parable to delta MEWS scores for prediction of death and ICU admission, and suggesting that the condition in which patients leave the ED is as important as changes while in the ED.

Supplementary data to this article can be found online at https://doi. org/10.1016/j.ajem.2019.12.024.

Ethics approval and consent to participate

This study was approved by the Institutional Review Board (IRB) under #00096120 on January 13th, 2017.

Consent for publication

Not applicable.

Availability of data and materials

To facilitate research reproducibility, replicability, accuracy and transparency, the datasets generated and/or analyzed during the

current study, and the associated analytic code, will be made available indefinitely [DOI 10.17605/OSF.IO/ESJ9K], following publication, to any- one who wishes to access the data for analysis, on the Open Science Foundation [20] (OSF) repository at [https://osf.io/esj9k/]. Data were de-identified in accordance with Section 164.514 of the Health Insur- ance Portability and Accountability Act (HIPAA).

Funding

JT was supported by a career development award (K23HL141596) from the National Heart, Lung, And Blood Institute (NHLBI) of the Na- tional Institutes of Health (NIH). This study was also supported, in part, by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant 5UL1TR001067-02 (formerly 8UL1TR000105 and UL1RR025764). The content is solely the responsi- bility of the authors and does not necessarily represent the official views of the National Institutes of Health. None of the funding sources were involved in the design or conduct of the study, collection, manage- ment, analysis or interpretation of the data, or preparation, review or approval of the manuscript.

Declaration of competing interest

The authors declare that they have no competing interests.

Acknowledgements

We would like to thank John Arego for his contributions to the devel- opment and implementation of the clinical MEWS alert program.

References

  1. Elixhauser A, Friedman B, Stranges E. Septicemia in U.S. hospitals, 2009: statistical brief #122. Healthcare cost and utilization project (HCUP) statistical briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006. http://www.ncbi. nlm.nih.gov/books/NBK65391/, Accessed date: 2 May 2019.
  2. Liu V, Lei X, Prescott HC, Kipnis P, Iwashyna TJ, Escobar GJ. Hospital readmission and healthcare utilization following sepsis in Community settings. J Hosp Med 2014;9

    (8):502-7. https://doi.org/10.1002/jhm.2197.

    Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 tRRough 2000. N Engl J Med 2003;348(16):1546-54. https://doi. org/10.1056/NEJMoa022139.

  3. Pfuntner A, Wier LM, Steiner C. Costs for hospital stays in the United States, 2010: statistical brief #146. Healthcare cost and utilization project (HCUP) statistical briefs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2006. http:// www.ncbi.nlm.nih.gov/books/NBK121966/. [Accessed May 2, 2019].
  4. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus def- initions for sepsis and septic shock (sepsis-3). JAMA 2016;315(8):801-10. https:// doi.org/10.1001/jama.2016.0287.
  5. Torio CM, Andrews RM. National inpatient hospital costs: the most expensive condi- tions by payer, 2011: statistical brief #160. Healthcare cost and utilization project (HCUP) statistical briefs. Rockville (MD): Agency for Healthcare Research and Qual- ity (US); 2006. http://www.ncbi.nlm.nih.gov/books/NBK169005/. [Accessed May 2, 2019].
  6. Paoli C, Reynolds M, Sinha M, Gitlin M, Crouser E. Epidemiology and costs of sepsis in the United States–an analysis based on timing of diagnosis and severity level*. Crit Care Med 2018;46(12):1889-97. https://doi.org/10.1097/CCM.0000000000003342.
  7. Filbin MR, Thorsen JE, Lynch J, et al. Challenges and opportunities for emergency de- partment sepsis screening at. Triage Sci Rep 2018;8(1):11059. https://doi.org/10. 1038/s41598-018-29427-1.
  8. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the emergency department. Am J Emerg Med 2018. https://doi.org/10.1016/j.ajem.2018.10.058 November.
  9. Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a Modified Early Warn- ing Score in medical admissions. QJM Mon J Assoc Physicians 2001;94(10):521-6.
  10. Delgado-Hurtado JJ, Berger A, Bansal AB. Emergency department modified early warning score association with admission, admission disposition, mortality, and length of stay. J Community Hosp Intern Med Perspect 2016;6(2):31456. https:// doi.org/10.3402/jchimp.v6.31456.
  11. Singer AJ, Ng J, Thode HC, Spiegel R, Weingart S. Quick SOFA scores predict mortality in adult emergency department patients with and without suspected infection. Ann Emerg Med 2017;69(4):475-9. https://doi.org/10.1016/j.annemergmed.2016.10. 007.
  12. Correia N, Rodrigues RP, Sa MC, Dias P, Lopes L, Paiva A. Improving recognition of pa- tients at risk in a Portuguese general hospital: results from a preliminary study on the early warning score. Int J Emerg Med 2014;7:22. https://doi.org/10.1186/ s12245-014-0022-7.
  13. Cameron A, Rodgers K, Ireland A, Jamdar R, McKay GA. A simple tool to predict ad- mission at the time of triage. Emerg Med J EMJ 2015;32(3):174-9. https://doi.org/ 10.1136/emermed-2013-203200.
  14. Brekke IJ, Puntervoll LH, Pedersen PB, Kellett J, Brabrand M. The value of vital sign trends in predicting and monitoring clinical deterioration: a systematic review. PLOS ONE 2019;14(1):e0210875. https://doi.org/10.1371/journal.pone.0210875.
  15. Kamikawa Y, Hayashi H. Predicting in-hospital mortality among non-trauma pa- tients based on vital sign changes between prehospital and in-hospital: an observa- tional cohort study. PLOS ONE 2019;14(1):e0211580. https://doi.org/10.1371/ journal.pone.0211580.
  16. Quinten VM, van Meurs M, Olgers TJ, Vonk JM, Ligtenberg JJM, Ter Maaten JC. Re- peated vital sign measurements in the emergency department predict patient dete- rioration within 72 hours: a prospective observational study. Scand J Trauma Resusc Emerg Med 2018;26(1):57. https://doi.org/10.1186/s13049-018-0525-y.
  17. Chiew CJ, Liu N, Tagami T, Wong TH, Koh ZX, Ong MEH. Heart rate variability based machine learning models for risk prediction of suspected sepsis patients in the emergency department. Medicine (Baltimore) 2019;98(6):e14197. https://doi.org/ 10.1097/MD.0000000000014197.
  18. von Elm E, Altman DG, Egger M, et al. The strengthening the reporting of observa- tional studies in epidemiology (STROBE) statement: guidelines for reporting obser- vational studies. PLoS Med 2007;4(10):e296. https://doi.org/10.1371/journal.pmed. 0040296.
  19. Foster ED, Deardorff A. Open science framework (OSF). J Med Libr Assoc 2017;105

    (2). https://doi.org/10.5195/JMLA.2017.88.

    Kawamoto K, Martin CJ, Williams K, et al. Value driven outcomes (VDO): a prag- matic, modular, and extensible software framework for understanding and improv- ing health care costs and outcomes. J Am Med Inform Assoc 2015;22(1):223-35. https://doi.org/10.1136/amiajnl-2013-002511.

  20. Lee VS, Kawamoto K, Hess R, et al. Implementation of a value-driven outcomes pro- gram to identify high variability in clinical costs and outcomes and association with reduced cost and improved quality. JAMA 2016;316(10):1061. https://doi.org/10. 1001/jama.2016.12226.
  21. Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS-towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010;81(8):932-7. https://doi.org/10.1016/j.resuscitation.2010.04.014.
  22. Textor J, van der Zander B, Gilthorpe MS, Liskiewicz M, Ellison GTH. Robust causal inference using directed acyclic graphs: the R package ‘dagitty. Int J Epidemiol Janu- ary 2017:dyw341. https://doi.org/10.1093/ije/dyw341.
  23. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci 2018;115(11):2600-6. https://doi.org/10.1073/pnas.1708274114.
  24. Jayasundera R, Neilly M, Smith TO, Myint PK. Are early warning scores useful predic- tors for mortality and morbidity in hospitalised acutely unwell older patients? A sys- tematic review. J Clin Med 2018;7(10). https://doi.org/10.3390/jcm7100309.
  25. Merz TM, Etter R, Mende L, et al. Risk assessment in the first fifteen minutes: a pro- spective cohort study of a simple physiological scoring system in the emergency de- partment. Crit Care Lond Engl 2011;15(1):R25. https://doi.org/10.1186/cc9972.
  26. Westphal GA, Pereira AB, Fachin SM, et al. An electronic warning system helps re- duce the time to diagnosis of sepsis. Rev Bras Ter Intensiva 2018;30(4):414-22. https://doi.org/10.5935/0103-507X.20180059.
  27. Peltan ID, Brown SM, Bledsoe JR, et al. ED door-to-antibiotic time and long-term mortality in Sepsis. Chest 2019. https://doi.org/10.1016/j.chest.2019.02.008 February.
  28. Kievlan DR, Zhang LA, Chang C-CH, Angus DC, Seymour CW. Evaluation of repeated quick sepsis-related organ failure assessment measurements among patients with suspected infection. Crit Care Med 2018;46(12):1906-13. https://doi.org/10.1097/ CCM.0000000000003360.
  29. Redfern OC, Smith GB, Prytherch DR, Meredith P, Inada-Kim M, Schmidt PE. A com- parison of the quick sequential (sepsis-related) organ failure assessment score and the national early warning score in non-ICU patients with/without infection. Crit Care Med 2018;46(12):1923-33. https://doi.org/10.1097/CCM.0000000000003359.
  30. Churpek MM, Snyder A, Han X, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195(7):906-11. https://doi.org/10.1164/rccm.201604-0854OC.
  31. Hwang SY, Jo IJ, Lee SU, et al. Low accuracy of positive qSOFA criteria for predicting 28-day mortality in critically ill septic patients during the early period after emer- gency department presentation. Ann Emerg Med 2018;71(1):1-9.e2. https://doi. org/10.1016/j.annemergmed.2017.05.022.
  32. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PWB. The impact of the use of the early warning score (EWS) on patient outcomes: a systematic review. Resuscitation 2014;85(5):587-94. https://doi.org/10.1016/j.re- suscitation.2014.01.013.
  33. Wuytack F, Meskell P, Conway A, et al. The effectiveness of physiologically based early warning or track and trigger systems after triage in adult patients presenting to emergency departments: a systematic review. BMC Emerg Med 2017;17(1):38. https://doi.org/10.1186/s12873-017-0148-z.

Leave a Reply

Your email address will not be published. Required fields are marked *