Article

A checklist manifesto: Can a checklist of common diagnoses improve accuracy in ECG interpretation?

a b s t r a c t

Objective: To determine whether a checklist of possible etiologies for syncope provided alongside ECGs helps Emergency Medicine (EM) residents identify ECG patterns more accurately than with ECGs alone. Methods: We developed a test of ten ECGs with syncope-related pathology from ECG Wave-Maven. We reviewed the literature and used expert consensus to develop a checklist of syncope-related pathologies commonly seen and diagnosed on ECGs. We randomized residents from three New York EM residency programs to interpret ECGs with or without a checklist embedded into the test.

Results: We randomized 165 residents and received completed tests from 100 (60%). Of those who responded, 39% were interns, 23% PGY2s, and 38% were PGY3s or PGY4s. We found no significant differ- ence in overall test scores between those who read ECGs with a checklist and those who read ECGs alone. In post-hoc analysis, residents given a checklist of syncoperelated etiologies were significantly more likely to recognize Brugada (96% vs. 78%, p = 0.007), long QT (86% vs. 68%, p = 0.03) and heart block (100% vs 78%, p = 0.003) as compared to those without a checklist. Those with a checklist were more likely to overread normal ECGs (72% vs 35%, p = 0.0001) compared to those without a checklist, finding pathology where there was none.

Conclusion: Using a checklist with common syncope-related pathology when interpreting an ECG for a patient with clinical scenario of syncope may improve residents’ ability to recognize some clinically important pathologies; however it could lead to increased interpretation and suspicion of pathology that is not present.

(C) 2019

Introduction

Background

Interpreting Electrocardiograms is a vital part of the practice of Emergency Medicine (EM). Patients presenting to the Emergency Department (ED) with syncope, chest pain, and other complaints relating to the cardiovascular system receive ECGs that must be interpreted rapidly and accurately [1]. ECGs are often pre- sented to the EM physician by a technician with limited Clinical context and as an interruption to other tasks, which can increase the likelihood of cognitive errors [2,3]. EM residents learn to inter- pret ECGs through lectures and Clinical training; however, there is no standard curriculum for teaching or evaluating their proficiency

Abbreviations: ECGs, electrocardiograms; EM, Emergency Medicine; ED, Emer- gency Department.

* Corresponding author at: 1249 Park Ave, Apt 5D, New York, NY 10029, United

States of America.

E-mail address: [email protected] (J. Nickerson).

[1]. To date, little research evaluates methods for reducing errors in ECG interpretation in EM.

Checklists have long been used to reduce error in fields such as airline travel. More recently, medicine has adopted checklists in an effort to improve safety [4]. However, the majority of studies have evaluated the use of checklists in procedural skills rather than cog- nitive skills.

There is a small body of literature demonstrating that checklists can reduce error in diagnosis. Diagnostic checklists help to broaden the differential by reminding physicians of all options to consider, rather than clinicians relying on memory and pattern recognition alone [5]. Research also demonstrates that checklists may be most useful for clinicians with less experience who have not yet develo- ped schemas for pattern recognition [6].

Importance

In the context of the ED, ECGs are often interpreted using pat- tern recognition by both novice and experienced ED physicians

https://doi.org/10.1016/j.ajem.2019.03.048

0735-6757/(C) 2019

[7]. This type of cognitive reasoning is especially susceptible to improved accuracy with the use of checklists for interpreting ECGs [5]. Improving accuracy in ECG interpretation is particularly important given the dire consequences of missed diagnoses.

Patients presenting to the ED with syncope represent a diagnos- tic challenge for EM physicians as the differential for syncope is broad and the diagnostic tests are poor. Cardiac syncope, accoun- ting for 15% of syncope cases [8] has been associated with a one- year mortality of up to 30% [9] and therefore is an essential diagno- sis in the ED. The ECG is a diagnostically useful test in the ED for cardiac syncope [10]. Therefore, enhancing training for EM resi- dents in diagnosing clinically important findings on ECG in Patients with syncope may represent an important area of research for reducing errors and improving patient outcomes.

Goals of this investigation

Our randomized control study sought to determine whether a checklist of common etiologies for syncope provided alongside ECGs helps EM resident physicians identify syncope-related ECG diagnoses more accurately than with ECGs alone.

Methods

Study design

We conducted a blinded randomized control trial of a conve- nience sample of residents from three New York City Emergency Medicine Residencies. All data were collected on a single day in April 2018.

Tool development

We reviewed the literature and used expert consensus to deve- lop a checklist of clinically important syncope-related pathology commonly seen and diagnosed on ECGs.

Experts consisted of practicing attending emergency physi- cians; they developed a list of relevant syncope-related diagnoses on ECG and reviewed the final list for inclusion in the study. (See Table 1)

Based on the checklist, we developed a test of ten ECGs with syncope-related pathology and clinical vignettes from ECG Wave- Maven [11]. ECG Wave-Maven is an internet-based electrocardio- graphy self-assessment program for clinicians that is open source [12].

Participants were prompted to provide their overall impression or diagnosis in a free response space. The test provided an example ECG showing an ST elevation myocardial infarction with a response of ”STEMI”. We chose this method to mimic the process of ECG evaluation that occurs in the clinical environment. Clinical vigne- ttes were succinct, such as, ”A 32 year old man presents with syn-

Table 1

List of Syncope-Related diagnoses seen on ECGs.

We developed a list of common and important syncope-related diagnoses seen on ECG by reviewing the literature and expert consensus. We provided this list with the test for those randomized to the checklist arm.

Below is a list of conditions which may cause syncope. You may refer to this list while you interpret the ECGs.

Brugada syndrome

Wolff-Parkinson-White syndrome Hypertrophic cardiomyopathy Long QT syndrome

Pericardial effusion Heart block Pulmonary embolism

Table 2

List of correct answers for ECG test.

List of correct diagnosis on ECG test

Q1 Brugada syndrome

Q2 Normal sinus

Q3 Hypertrophic cardiomyopathy

Q4 Long QT syndrome

Q5 Pericardial effusion

Q6 Normal sinus

Q7 Heart block

Q8 Pulmonary embolism

Q9 Heart block

Q10 Wolff-Parkinson-White syndrome

The table shows the correct answers for each question of our ten question ECG quiz on syncope related pathology.

cope”. We did not provide additional information to mimic the manner in which ECGs are presented clinically. We piloted the test with and without the checklist on medical students, junior, and senior EM residents who would not be in the study cohort to assess for readability and difficulty. Small alterations were made to the wording and order of the test based on feedback from our pilot par- ticipants. (See Table 2)

Participant selection

All residents from the three Emergency Medicine residencies were eligible to participate. Those who elected not to participate were not included.

Randomization

We randomized residents from three New York EM residencies using a sealedenvelope.com blocked randomization scheme to interpret ECGs with or without a checklist embedded into the test. Our randomization scheme included block sizes of 4, 6, and 8. Resi- dents were not told about the randomization or possibility of recei- ving the checklist, but simply asked to complete the test emailed to them.

We sent the test by email during dedicated conference time and residents completed the test on their electronic devices within 10 min. Participation was voluntary and anonymous. (See Fig 1)

Score calculation

Two EM trained physicians, blinded to participant group, reviewed responses and designated them as correct or incorrect. For example, if the correct answer was ”heart block” responses such as ”heart block” and ”3rd degree heart block” and ”AV nodal block” were all recorded as correct. The two reviewers agreed on all responses.

Outcome measurements

The primary outcome was total score compared between the two groups. We also evaluated scores on individual questions between the two groups.

Statistical analysis

We calculated a sample size of 49 participants in each arm to detect a 22% difference in mean test scores as we determined that a greater than two question difference in scores would be a mea- ningful difference. Dichotomous variables were analyzed using Fis- her’s exact tests using a two-tailed p-value.

Ethical approval was obtained by the institutional review board.

Fig. 1. The CONSORT diagram details participants who were eligible, randomized, and analyzed in this randomized controlled trial.

Results

Characteristics of study participants

Among the three residencies, 165 residents were eligible to par- ticipate. Of those, 100 (60%) agreed to participate and submitted completed tests. We analyzed data from all 100 residents. Of par- ticipants, 39% were interns, 23% PGY2s, and 38% were PGY3s or PGY4s.

Primary outcome

We found no significant difference in overall test scores between those who read ECGs with a checklist and those who read ECGs alone. Those with the checklist had median score of 7.2 (stan- dard deviation (SD) 1.4) out of 10 while those without the checklist had a median score of 6.8 (SD1.6) (p = 0.19). (See Fig 2)

Secondary outcomes

Fig. 2. Total score by randomization group. This graphic shows there was no difference total scores stratified by group.

In post-hoc analysis, residents given a checklist of syncope- related etiologies were significantly more likely to recognize Bru- gada (96% vs. 78%, p = 0.007), long QT (86% vs. 68%, p = 0.03) and heart block (100% vs 78%, p = 0.003) as compared to those without a checklist. Overall recognition for hypertrophic cardiomyopathy (HOCM) was poor, but higher in those given a checklist (26% vs 6%, p = 0.005). Those with a checklist were more likely to overread

normal ECGs (72% vs 35%, p = 0.0001) compared to those without a

Table 3

Impact of checklist by pathology.

Checklist (N = 50) % correct (frequency)

No checklist (N = 50%) correct (frequency)

p-value

checklist, finding pathology where there was none. There was no significant difference in diagnostic accuracy between the groups in ECGs with signs of PE, WPW or pericardial effusion. (See Table 3)

We did not find any significant differences when stratifying by level of training.

Discussion

Our results are surprising in that we did not find a significant difference in total score between those with and those without a

Brugada 96% (48) 78% (39) 0.007

HOCM* 26% (13) 6% (3) 0.006

Long QT 86% (43) 68% (34) 0.03

Heart Block 99% (99) 85% (85) 0.003

Normal Sinus 35% (35) 72% (72) 0.0001

This chart shows the difference in identification of specific pathologies stratified by those with a checklist and without a checklist. It demonstrates that those with a checklist were more likely to identify Brugada, HOCM, Long QT, and Heart Block compared to those without a checklist. It also shows that those without a checklist more commonly correctly answered normal sinus for EKGs that did not contain pathology compared to those with a checklist.

*HOCM: Hypertrophic Cardiomyopathy.

checklist. This is in contrast to prior research that demonstrates that use of checklists improves competency in interpreting ECGs [5,13].

However, our results do show a higher rate of recognition of potentially fatal channelopathies such as Brugada and long QT by residents given the checklist compared to those without a che- cklist. These diagnoses are rare and may have been more easy to identify with a checklist that reminded clinicians of a broad diffe- rential for cardiac syncope when reviewing the ECGs. The role of diagnostic checklists helping physicians to maintain a broad diffe- rential has been demonstrated in previous literature [14-16]. This finding is clinically important and demonstrates the potential for checklists to provide a benefit in the clinical context and the potential to improve patient outcomes with the use of checklists [17].

Our results also demonstrate that those given the checklist were more likely to diagnose pathology where there was none. This phenomenon was seen in a study of EM and internal medicine residents given ECGs with or without erroneous computer inter- pretations of ischemia. In this study, residents who were given the erroneous computer interpretation were more prone to acting on ”ischemia” even though it was not present on the ECG [18].

Interestingly, we did not find significantly different rates of cor- rect diagnoses between those with a checklist and those without for ECGs with PE, WPW, or pericardial effusion. We hypothesize that these diagnoses are more commonly taught about and dis- cussed than the others.

We were also surprised to find no statistical difference in results when participants were stratified by level of training. Prior research has supported the idea that diagnostic aids may dispro- portionately impact the practice of physicians with less training [6]. However, other research has contradicted this idea by showing that physicians with more training are more susceptible to prema- ture diagnostic closure and may be at higher risk of failing to con- sider alternative diagnosis [19]. Given our small sample size, this lack of difference may be due to a lack of power for adequate ana- lysis of subgroups. Further studies should evaluate the differing impact of checklists by year of training.

Our results are important because syncope is a common EM presentation and missing clinically important ECG pathology can have dire consequences. Thus, research that illuminates methods for reducing errors could have a significant impact on patient outcomes. Additionally, ECG training and competency remains an understudied and under-evaluated area of EM training [1,20].

Limitations

Our study was limited to a convenience sample of residents from three emergency medicine residencies in one geographic area. The presentation and wording of the quiz and checklist may have influenced those with the checklist to choose an answer from the list (similar to multiple choice options) rather than using it as a guide and allowing for the possibility of a nor- mal ECG. Additionally, as ECG Wave-Maven is a free public site, some residents may have viewed the ECGs we used in this study before taking the test.

Our checklist was developed by a review of the literature and an expert consensus but does not include all pathology related to syn- cope and therefore is not comprehensive. Finally, the test was pro- vided outside of the clinical context and therefore cannot evaluate how the checklist could help physicians within the ED or how the checklist could affect practice or clinical outcomes.

Conclusion

When available, a checklist detailing common syncope-related pathologies may assist residents in the interpretation of syncope patients’ ECGs. However, this tool might lead to increased reading of pathology that is not present.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declarations of interest

None.

Presentation

This research was presented in oral format at ACEP 2018, San Diego, CA.

Acknowledgements

We would like to thank the residents from Mount Sinai, Mount Sinai St. Luke’s and Mount Sinai Beth Israel for their participation in this study.

References

  1. Ginde AA, Char DM. Emergency medicine residency training in electrocardiogram interpretation. Acad Emerg Med 2003 Jul;10(7):738-42.
  2. Skaugset LM, Farrell S, Carney M, Wolff M, Santen SA, Perry M, et al. Can you multitask? Evidence and limitations of task switching and multitasking in emergency medicine. Ann Emerg Med 2016 Aug;68(2):189-95. https://doi. org/10.1016/j.annemergmed.2015.10.003 [Epub 2015 Nov 14].
  3. Croskerry P. Cognitive forcing strategies in clinical decision making. Ann Emerg Med 2003 Jan;41(1):110-20.
  4. Gawande Atul. A checklist manifesto: How to get things right, 2011.
  5. Sibbald M, de Bruin AB, van Merrienboer JJ. Checklists improve experts’ diagnostic decisions. Med Educ 2013 Mar;47(3):301-8. https://doi.org/ 10.1111/medu.12080.
  6. Sibbald M, De Bruin AB, van Merrienboer JJ. Finding and fixing mistakes: do checklists work for clinicians with different levels of experience? Adv Health Sci Educ Theory Pract 2014 Mar;19(1):43-51. https://doi.org/10.1007/s10459- 013-9459-3 [Epub 2013 Apr 27].
  7. Norman G, Young M, Brooks L. Non-analytical models of clinical reasoning: the role of experience. Med Educ 2007 Dec;41(12):1140-5 [Epub 2007 Nov 13].
  8. Anderson J, O’Callaghan P. Cardiac syncope. Epilepsia 2012 Dec;53(Suppl. 7):34-41.
  9. Guimaraes RB, Essebag V, Furlanetto M, Yanez JPG, Farina MG, Garcia D, et al. Structural heart disease as the cause of syncope. Braz J Med Biol Res 2018 Mar 01;51(4):e6989.
  10. Mizrachi EM, Sitammagari KK. Cardiac syncope. Treasure Island (FL): StatPearls Publishing; 2018.
  11. Nathanson LA, McClennen S, Safran C, Goldberger AL. ECG wave-maven: Self- assessment program for students and clinicians, http://ecg.bidmc.harvard.edu.
  12. McClennen S, Nathanson LA, Safran C, Goldberger AL. ECG wave- maven: An internet-based electrocardiography self-assessment program for students and clinicians, vol. 8: 2. Med Educ Online; 2003. Available from http://www.med- ed-online.org [serial online].
  13. Talebian MT, Zamani MM, Toliat A, Ghasemzadeh R, Saeedi M, Momeni M, et al. Evaluation of emergency medicine residents competencies in electrocardiogram interpretation. Acta Med Iran 2014;52(11):848-54.
  14. Ely JW, Graber ML, Croskerry P. Checklists to reduce diagnostic errors. Acad Med 2011 Mar;86(3):307-13. https://doi.org/10.1097/ ACM.0b013e31820824cd.
  15. Ely JW, Graber MA. Checklists to prevent diagnostic errors: a pilot randomized controlled trial. Diagnosis (Berl) 2015 Sep 1;2(3):163-9. https://doi.org/ 10.1515/dx-2015-0008.
  16. Graber ML, Sorensen AV, Biswas J, Modi V, Wackett A, Johnson S, et al. Developing checklists to prevent diagnostic error in emergency room settings. Diagnosis (Berl) 2014 Sep;1(3):223-31. https://doi.org/10.1515/dx-2014-0019 [Epub 2014 Jun 19].
  17. Chew KS, Durning SJ, van Merrienboer JJ. Teaching metacognition in clinical decision-making using a novel mnemonic checklist: an exploratory study.

    Singapore Med J 2016;57(12):694-700. https://doi.org/10.11622/ smedj.2016015.

    Southern WN, Arnsten JH. The effect of erroneous computer interpretation of ECGs on resident decision making. Med Decis Making 2009 May-Jun;29

    (3):372-6. https://doi.org/10.1177/0272989X09333125 [Epub 2009 May 21].

    Eva K, Cunnington J. The difficulty with experience: does practice increase susceptibility to premature closure. J Contin Educ Heal Prof 2006;26:192-8.

  18. Salerno SM, Alguire PC, Waxman HS. Competency in interpretation of 12-lead electrocardiograms: a summary and appraisal of published evidence. Ann Intern Med 2003 May 6;138(9):751-60.

Leave a Reply

Your email address will not be published. Required fields are marked *