Cardiology

Comparing physicians and experienced advanced practice practitioners on the interpretation of electrocardiograms in the emergency department

a b s t r a c t

Background: Many patients present to emergency departments (ED) in U.S. for evaluation of acute coronary syn- drome, and a rapid electrocardiogram (ECG) and interpretation are imperative for initial triage. A growing num- ber of advanced practice practitioners (APP) (e.g. physician assistants, nurse practitioners) are assisting patient care in the ED.

Purpose: This study aims to compare the interpretation of ECGs by experienced APPs, each having 10 or more years of experience, with resident physicians and attending physicians.

Patients and methods: 99 ED providers were stratified into attendings, residents at varying levels, and APPs were tested to interpret 36 ECGs from a database of ECGs initially interpreted to be ST elevation myocardial infarctions, of which 24 were determined to have a culprit lesion by coronary intervention.

Results: Attending physicians were the most sensitive (0.86, 95% CI of 0.80 to 0.92) and specific (0.69, 95% Cl of 0.60 to 0.79) at interpreting ECGs, but APPs and physicians in their first year of practice out of residency were al- most equally as sensitive [(0.82, 95% CI of 0.76 to 0.88) and (0.82, 95% CI of 0.76 to 0.88)] and specific [(0.62, 95% cl of 0.52 to 0.73) and (0.65, 95% Cl of 0.56 to 0.75)].

Conclusion: This study suggests the possibility of changing ED workflow where experienced APPs can be respon- sible for initial screening of an ECG, thus allowing fewer interruptions for ED physicians.

(C) 2020

  1. Introduction/background

A large number of patients present to emergency departments (ED) in U.S. for evaluation of acute coronary syndrome (ACS). For these pa- tients, a rapid electrocardiogram (ECG) and physician interpretation within 10 min of ED arrival is recommended by the American Heart As- sociation and American College of Cardiology [1]. Increasingly, hospitals are employing advanced practice practitioners (e.g. physician assistants, nurse practitioners) to assist in patient care at points of triage as well as in the main ED. These providers may be placed in a position to review screening ECGs. False negative interpretation of an ECG showing ST ele- vation myocardial infarction (STEMI) carries significant risk of morbid- ity and mortality, whereas false positive interpretation can lead to inappropriate resource utilization and morbidity due to iatrogenic com- plications [1]. ED physician and resident accuracy in the evaluation of identifying STEMI has been extensively published [2,3]. However, there is limited data on the accuracy of ED advanced practice

* Corresponding author at: 1411 E 31st St, Oakland, CA 94602, United States of America.

E-mail addresses: [email protected] (A. Hoang), [email protected] (A. Singh), [email protected] (A. Singh).

practitioners (APPs) in this area. In this cross-sectional survey we ana- lyze the diagnostic accuracy and agreement of ECG interpretation by emergency medicine providers, including physicians with varying levels of experience and APPs.

  1. Methods
    1. Data source

A previous study on accuracy of physician recognition of STEMI ECGs was used as the basis for this study. All of the EKGs are standard 12 lead EKGs on Philips Pagewriter Machine on 25 mm paper speed. The afore- mentioned study’s authors selected 36 ECG’s from a large registry of clinical STEMI diagnoses where cardiac catheterization outcomes were known (full 12-lead ECGs provided in the supplemental material for the original study). These STEMI diagnoses were all made primarily by ED physicians, only with consultation from cardiology at the discretion of the ED physician on the uncertainty of whether the patient needed emergent PCI. The total number of ECGs chosen was based on physician pooling regarding the number of ECGs each physician would be willing to read without compensation [3]. ECGs demonstrating isolated Left bundle branch block or ventricular rhythms were excluded, as were

https://doi.org/10.1016/j.ajem.2020.01.047

0735-6757/(C) 2020

146 A. Hoang et al. / American Journal of Emergency Medicine 40 (2021) 145147

ECGs from patients who did not undergo emergency angiography. Pa- tient identifiers were removed from the ECG, as well as age and gender identifiers. Within this group of 36 ECGs, twelve (33%) were considered “false positive” ECGs; i.e. no culprit lesion identified at time of cardiac catheterization and all vessels were reported to have Thrombolysis in Myocardial Infarction 3 flow. The remaining 24 ECGs all had find- ings consistent with ST-segment elevation myocardial infarction.

    1. Participants

ED attendings, residents, and advanced practice practitioners from a community teaching hospital that serves as a primary receiving center for suspected STEMI patients, formed the basis of the study population. Experience of medical providers consisted of emergency medicine resi- dents in training (stratified by post graduate year [PGY] training level), 1st year emergency medicine residency graduates (including 1st year fellows of an emergency medicine ultrasound fellowship) [PGY5s], ad- vanced practice practitioner in emergency medicine, and emergency medicine attending physician. APPs with over 10 years of clinical expe- rience in emergency department work in our main ED as well as our ur- gent care area, and are experienced in reading ECGs. ECGs were distributed to providers electronically or by paper. Participation was strongly encouraged but no reward or punishment was associated with participation.

Each study participant was given the same 36 ECGs, in the same order. Participants were instructed to decide, based on the ECG alone, if the patient met STEMI criteria and whether immediate cardiac cathe- terization was indicated. Participants were only told that all ECGs were from patients with equal risk of acute myocardial infarction. Partici- pants were not aware that all patients with ECGs included in the study went for emergent cardiac catheterization [4]. Participants were given 1 h from the time they received the ECG packet, to submit all their re- sponses. Any ECGs with omitted response were re-sent to the study par- ticipant for their response to the study question.

    1. Definitions

In the absence of an alternative confounding ECG pattern (e.g. LBBB, LVH with strain, pericarditis, etc.), STEMI is typically identified as new ST segment elevation at the J point of at least 1 mm in two contiguous leads in the limb leads, with 1.5 mm, 2 mm, or 2.5 mm used as the min- imal amount of ST segment elevation in the anterior precordial leads de- pending upon age and gender of the patient [5]. ST depressions in V1-V3 with upright T waves and a R/S ratio N1 in V1 was regarded to be consis- tent with isolated posterior STEMI. True positive ECGs were defined as an ECG that led to the identification of a culprit coronary lesion or at least 1 coronary artery with less than TIMI 3 blood flow. False positive ECGs were defined as no thromboembolic lesion or at least TIMI 3 blood flow in all coronary arteries.

    1. Statistical analysis

Each result is binary and defined as accurate or inaccurate. Results were stratified into cohorts based on years of experience and level of training. Per cohort, the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), Fleiss kappa statistic, and 95% confidence interval were calculated [3].

  1. Results

In total, 99 ED providers and medical participated in this study. Participants included 18 attendings, 10-12 physicians from each post graduate year 1 through 5, and 19 experienced advanced practice practitioners.

Cumulatively, participants identified a true STEMI with 76.9% sensi- tivity (95% CI of 74.4 to 79.4) and 65% specificity (95% CI of 61.7 to 68.5).

Table 1

Provider accuracy in interpreting ECGs by experience and/or role.

Sensitivity

Specificity

PPV

NPV

Kappa

Overall group

76.90%

65.00%

80.3%%

64.90%

0.31

ATT

86.00%

68.80%

84.50%

76.60%

0.42

PGY5

81.70%

65.40%

81.40%

69.10%

0.43

PGY4

75.70%

67.70%

81.50%

62.80%

0.45

PGY3

70.40%

74.60%

83.10%

60.00%

0.37

PGY2

76.80%

61.50%

79.10%

63.50%

0.29

PGY1

64.50%

58.30%

73.00%

49.50%

0.21

APPs

81.90%

62.30%

80.80%

71.50%

0.08

The PPV or the probability that a provider decides an ECG meets STEMI criteria and identifies an intervenable lesion is 80.3% (95 CI of 78.8 to 81.8). The NPV or the probability that a provider decides that an ECG does not meet STEMI criteria and there is no intervenable lesion is 64.9% (95% CI of 62.1 to 67.8). Amongst all providers, we identified a Fleiss kappa statistic of 0.314 where 1 indicates perfect Interobserver agreement and -1 indicated perfect interobserver disagreement.

The main outcome of interest is the sensitivity of providers screening ECGs. ED attendings were the most sensitive at 86.0% (95% CI of 79.7 to 91.5). First year graduates from residency (PGY5) and APPs are the next most sensitive and accurate with sensitivities of 81.7% (95% CI of 76.1 to 87.8) and 81.9% (95% CI of 75.5 to 87.9).

Fleiss kappa statistics for attendings and PGY5s showed the most interprovider agreement with kappas of 0.42 and 0.43. Amongst all the cohorts, APPs represented the most inconsistencies with their inter- pretation with a kappa of 0.08.

As seen in Table 2, PGY5 and APPs had a 80% (OR 0.08: Cl 0.60-1.05) and 75% (OR 0.75: CI 0.57-1.00) chance respectively of accurately interpreting ECGs when compared to attendings. Junior and senior res- idents (PGY2-4) had similar chances of accurately interpreting ECGs compared to attendings with 68% (OR 0.68: Cl 0.52-0.88), 65% (OR 0.65, Cl 0.49-0.87), and 63% (OR 0.63: Cl 0.48-0.83) respectively.

Residents in their 3rd or 4th year of residency often graduate to a level where they can screen ECGs. As seen in Table 3 and using PGY3- PGY4 as reference, we can see that the odds of accurately interpreting the EKG are 51% more in the attending group than the PG3-PGY4 group (OR 1.51; CI 1.18-1.92). PGY5 are 12% more likely to accurately interpret the ECG (OR 1.12; CI 0.96-1.50). Finally APPs are 14% more ac- curate than PGY3-PGY4 (OR 1.14; CI 0.91-1.42) at interpreting EKGs.

  1. Discussion

Our study demonstrates that physician sensitivity and accuracy for STEMI recognition on ECG improves with increasing level of training. Attending physicians had the highest sensitivity, specificity, accuracy, and Interrater agreement in detection of true positive and negative ECGs for STEMI (Table 1). Attending ECG interpretation had a moderate level of agreement (? = 0.42). ED residents showed a trend towards im- proving accuracy in detection of true positive and negative ECGs, and interrater agreement with higher level of training (Table 2). These find- ings are consistent with prior literature in the field, which seems to sug- gest better sensitivity and accuracy in ECG interpretation for STEMI with increasing education and experience [2,3].

Table 2

Comparison of ECG reading accuracy by role versus attendings.

Roll

Odds ratio

95% CI

P value

Attendings

REF

APPs

0.75

0.57-1.00

0.05

PGY5

0.80

0.60-1.05

0.11

PGY4

0.68

0.52-0.88

b0.01

PGY3

0.65

0.49-0.87

b0.01

PGY2

0.63

0.48-0.83

b0.01

PGY1

0.42

0.31-0.56

b0.01

A. Hoang et al. / American Journal of Emergency Medicine 40 (2021) 145147 147

Table 3

Comparison of ECG reading accuracy by role versus residents (PGY3-PGY4).

Roll

Odds ratio

95% CI

P value

Residents (PGY3-4)

REF

Attendings

1.51

1.18-1.92

b0.01

APPs

1.14

0.91-1.42

0.26

PGY5

1.12

0.96-1.50

0.11

PGY4

1.02

0.83-1.25

0.84

PGY3

0.98

0.78-1.24

0.83

PGY2

0.95

0.77-1.18

0.63

PGY1

0.63

0.50-0.79

b0.01

An important finding in our study is the relatively high sensitivity and accuracy of experienced APPs for STEMI recognition. APP ECG inter- pretation sensitivity and accuracy rivals that of graduating ED residents and ED fellows in their 5th year of postgraduate education. While APPs demonstrate high sensitivity, this may come at the expense of increase in false positive diagnosis of STEMI as reflected in APPs lower specificity and interrater agreement in ECG diagnosis of STEMI. This is most likely due to APPs lower risk aversion and experience reading ECGs given the setting of their practice, which is often in urgent care. Group and targeted ECG education for specific providers may be needed to address these issues.

This study may have important implications for existing ED ECG in- terpretation workflows, which largely rely on ED attending physicians to screen ECGs for STEMI. While rapid ECG interpretation is crucial for recognition of time sensitive diagnosis of STEMI, it also creates disrup- tions in usual ED physician workflow that can lead to important distrac- tion mediated adverse events like Medication errors and failure to address Abnormal vital signs of other unstable patients [6,7]. Increasing system reliance on computer interpretation of ECGs has been suggested as a solution to help minimize interruption of ED physician tasks. How- ever, the software used for ECG interpretation varies between different sites and currently human ECG interpretation for STEMI outperforms software/machine based interpretation [8]. New ECG screening workflows that maintain human ECG interpretation and decrease ED at- tending interruption may be desirable to avoid interruption associated errors and computer misinterpretation of STEMI ECGs. Presuming that ECG interpretation sensitivities of physicians in their PGY4 & 5 are ac- ceptable, we can imagine a system where experienced APPs could be the first level of screening. However, this comes at the discretion of in- dividual physician groups to accept the risk of lower sensitivities com- pared to more experienced attendings.

If additional studies in other practice settings confirm the results of

this study, an alternative workflow can be imagined where ECGs are ini- tially screened by the appropriate APPs and any Abnormal ECGs are im- mediately forwarded to the ED physician for review. This simple change in the workflow could decrease physician interruptions for ECG screen- ing without compromising on the sensitivity for STEMI detection. Addi- tionally this could address the issue of false positive ECG diagnosis of STEMI and poor APP specificity through immediate attending review of ECGs deemed concerning for STEMI by APP.

Our study has a number of strengths. All ECGs included in the study are from real STEMI activations that had diagnostic angiography

performed. The study included APPs and physicians from different levels of training with all subjects interpreting the same ECGs. We ac- knowledge certain limitations of this study which should be considered when interpreting the results. First, this was a single center study with an active cardiac catheterization laboratory, an emergency medicine residency program, and routine APP staffing in the high Acuity area of ED. These environmental features are not present across the majority of EDs, and thus limit the external generalizability of the results. Fur- thermore, the majority of our APPs have over 10 years of ED experience and thus may be better skilled at picking up STEMI cases than less expe- rienced APPs. Additionally, the overall rates of sensitivity, specificity, and accuracy in our cohort may be lower than expected based on the lack of clinical information provided for each patient and ECG. This study does not accurately reflect the reality of an ED where physicians and APPs may have different abilities to incorporate pretest probability into their interpretation.

Sources of support

Highland Hospital Alameda County for equipment.

CRediT authorship contribution statement

Alexander Hoang: Writing – original draft, Writing – review & editing, Visualization, Formal analysis. Amarinder Singh: Validation, Project administration, Investigation. Amandeep Singh: Supervision, Conceptualization, Data curation, Methodology.

References

  1. Levine GN, Bates ER, Blankenship JC, et al. 2015 ACC/AHA/SCAI focused update on pri- mary percutaneous coronary intervention for patients with ST-elevation myocardial infarction: an update of the 2011 ACCF/AHA/SCAI guideline for percutaneous coro- nary intervention and the 2013 ACCF/AHA guideline for the management of ST- elevation myocardial infarction: a report of the American College of Cardiology/ American Heart Association Task Force on Clinical Practice Guidelines and the Society for Cardiovascular Angiography and Interventions. Circulation 2016;133(11): 1135-47.
  2. Veronese G, Germini F, Ingrassia S, et al. Emergency physician accuracy in interpreting electrocardiograms with potential ST-segment elevation myocardial infarction: is it enough? Acute Card Care 2016;18(1):7-10.
  3. McCabe JM, Armstrong EJ, Ku I, et al. Physician accuracy in interpreting potential ST- segment elevation myocardial infarction electrocardiograms. J Am Heart Assoc 2013; 2(5):e000268.
  4. Cruz MF, Edwards J, Dinh MM, Barnes EH. The effect of clinical history on accuracy of electrocardiograph interpretation among doctors working in emergency depart- ments. Med J Aust 2012;197:161-5.
  5. Wagner GS, Macfarlane P, Wellens H, Josephson M, Gorgels A, Mirvis DM, et al. AHA/ ACCF/HRS recommendations for the standardization and interpretation of the electro-cardiogram. J Am Coll Cardiol 2009;53:1003-11.
  6. Pham JC, Story JL, Hicks RW, et al. National study on the frequency, types, causes, and consequences of voluntarily reported emergency department medication errors. J Emerg Med 2011;40(5):485-92.
  7. Westbrook JI, Woods A, Rob MI, Dunsmuir WTM, Day RO. Association of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med 2010;170(8):683-90.
  8. Mawri S, Michaels A, Gibbs J, et al. The comparison of physician to computer interpreted electrocardiograms on ST-elevation myocardial infarction door-to- balloon times. Crit Pathw Cardiol 2016;15(1):22-5.

Leave a Reply

Your email address will not be published. Required fields are marked *