Article, Radiology

Interrater reliability of pediatric point-of-care lung ultrasound findings

a b s t r a c t

Objective: We sought to assess interrater reliability (IRR) of lung Point-of-care ultrasound find- ings among pediatric patients with suspected pneumonia.

Methods: A convenience sample of patients between the ages of 6 months and 18 years with a clinical suspicion of pneumonia had a lung ultrasound performed by a POCUS-credentialed emergency medicine physician with subsequent expert review. Each lung zone was assessed as either normal or abnormal, and specific ultrasound findings were recorded. IRR was assessed by intraclass correlation coefficient (ICC) and kappa statistics.

Results: Seventy-one patients, with a total of 852 lung zones imaged, were included. The sonographer assessment of normal versus abnormal, across each of the zones, demonstrated moderate agreement with ICC 0.46 (95% CI: 0.41, 0.52) and kappa 0.56. Right-sided zones demonstrated moderate agreement [0.43 (CI 0.35, 0.51)] while left-sided zones, specifically left-sided anterior zones, showed only fair agree- ment [0.36 (0.28, 0.44)]. IRR varied between specific findings: ICC for B-lines 0.52 (95% CI: 0.46, 0.57),

pleural effusion 0.40 (0.34, 0.45), consolidation 0.39 (0.33, 0.44), subpleural consolidation 0.31 (0.25, 0.37), and pleural line irregularity 0.16 (0.10, 0.23). A composite indicator of typical pneumonia findings (consolidation, B-lines, and pleural effusion) demonstrated moderate [ICC 0.52 (0.46, 0.57)] reliability. Conclusions: We found moderate interrater reliability of lung POCUS findings for the assessment of pedi- atric patients with suspected pneumonia. B-lines had the highest reliability. Further assessment of lung POCUS is necessary to guide proper training and optimal scanning techniques to ensure adequate relia- bility of ultrasound findings in the assessment of pediatric pneumonia.

(C) 2019

Introduction

Point-of-care ultrasound of the lung has been shown to be accurate for the diagnosis of pediatric and adult community- acquired pneumonia (CAP) [1-4]. Several recent studies and meta-analyses support the utility of point-of-care lung ultrasound for pediatric CAP when compared to chest radiography [3-9]. Despite this growing evidence for lung POCUS and the limited value of routine physical examination findings for pediatric pneu- monia [10,11], the implementation of lung POCUS into pediatric clinical practice has been limited due to a reliance on chest radio-

* Corresponding author at: Boston Children’s Hospital, Division of Emergency Medicine, BCH 3066, 300 Longwood Ave, Boston, MA 02115, United States of America.

E-mail addresses: [email protected] (C.A. Gravel), michael. [email protected] (M.C. Monuteaux), jason.levy@childrens. harvard.edu (J.A. Levy), [email protected] (A.F. Miller), [email protected] (R.L. Vieira), richard.bachur@childrens. harvard.edu (R.G. Bachur).

graphs, the unavailability of POCUS in frontline practice settings, and the relative sonographic inexperience of many practitioners.

When considering POCUS for the evaluation of pneumonia, there is significant overlap of lung ultrasound findings encountered across several pediatric lung pathologies, including pneumonia, bronchiolitis, and reactive airway disease [12-16]. Heterogeneity exists across studies regarding which ultrasound findings are diag- nostic for pediatric pneumonia; a recent meta-analysis demon- strated variability in the accuracy of lung POCUS based on which findings or combination of findings are considered diagnostic [4]. Additionally, the clinical significance of some of these findings (e.g. subpleural consolidations) has not yet been clearly defined. standardized definitions of lung ultrasound artifacts and findings have been published in Consensus guidelines, [17] but there has been little comprehensive assessment of the reliability of these findings. It is imperative to understand the reliability of ultrasound findings before routinely utilizing lung POCUS for the evaluation of pediatric patients with acute respiratory complaints and possible pneumonia.

This study sought to assess interrater reliability (IRR) of point-

of-care lung ultrasound findings in pediatric patients with acute

https://doi.org/10.1016/j.ajem.2019.01.047

0735-6757/(C) 2019

respiratory complaints. As secondary analyses, we evaluated IRR for anatomic zone groupings, by patient age, by sonographer expe- rience, for individual lung ultrasound abnormalities, and as a com- posite of findings for pneumonia.

Methods

Study design, setting, and population

A convenience sample of pediatric patients with acute respira- tory complaints and possible pneumonia were enrolled from March 2017 to June 2018. Patients between the ages of 6 months and 18 years who presented to a single tertiary care pediatric emergency department (ED) with acute respiratory complaints were assessed for eligibility. Patients were identified by research coordinators based on chief complaint (e.g. fever, cough, respira- tory distress, chest pain) on an electronic tracking board during times in which a study sonographer was available. The treating emergency medicine attending was approached after the initial clinical evaluation to determine if the patient was being evaluated for pneumonia with a chest radiograph. The acquisition of the chest radiograph provided a clear indicator of suspected pneumo- nia and provided a realistic study population to assess future lung POCUS utilization in the ED setting.

If a radiograph was planned, caregivers were approached for con-

sent and assent was obtained from patients when appropriate. Once enrolled, a study sonographer was contacted to complete the study ultrasound. An image acquisition protocol was followed and is described in detail below. For purposes of this study, a second, inde- pendent and blinded expert sonographer reviewed the digitally- recorded video clips obtained by the primary sonographer. Patients were excluded from our analysis if a chest radiograph was not ulti- mately performed or if the sonographer became aware of the chest radiograph results before performance of the study ultrasound. The hospital institutional review board approved this study.

Sonographers

Our seven study sonographers were comprised of two groups: 1) four board-certified pediatric emergency medicine (PEM) physi- cians who had completed Registered Diagnostic Medical Sonogra- pher (RDMS) certification and/or a formal PEM POCUS fellowship, considered ”expert sonographers” and 2) three board-certified PEM physicians who were credentialed by our department in Lung point-of-care ultrasound and had completed >50 lung ultrasounds prior to enrollment, considered ”intermediate sonographers”. As part of the lung ultrasound research team, all sonographers com- pleted an additional one-hour didactic session led by the principal investigator (PI), consisting of a review of lung ultrasound artifacts and findings, the definitions and appearance of these findings based on consensus guidelines, common pitfalls and potential errors, and the study scanning protocol described below. Each sonographer then passed a knowledge-assessment evaluation, was given direct feedback for any errors, and performed a proctored lung ultrasound with the PI to ensure compliance with and consistency of the study scanning protocol. Each sonographer was given direct feedback on their initial study ultrasounds after review for quality assurance.

Study scanning protocol

Our study scanning protocol was previously established in the literature [18-21] and similar to the protocol initially described by Copetti and Cattarossi [22]. The protocol involved scanning 6 zones in each hemithorax, consisting of superior and inferior zones along each of the mid-clavicular (anterior), mid-axillary (lateral) and mid-scapular (posterior) lines. Both transverse and longitudi- nal video clips were saved in each of the 12 zones. A minimum of 24 six-second clips were saved for each patient, 2 in each zone,

with additional clips saved if necessary to ensure capture of the representative findings in each zone. All study ultrasounds were performed on a TE-7 Mindray machine (Mindray, Shenzhen, China) with a 12-4 mHz linear transducer at a default depth of 5 cm. Clips were saved to a web-based archiving system, Q-path (Telexy Healthcare, British Columbia, Canada), for review.

Measurements and outcome measures

Immediately after completion of the study ultrasound, the sonographer completed a data collection instrument to identify each zone as normal or abnormal. If a zone was considered abnor- mal, the sonographer recorded which of the following specific find- ings were encountered: absence of lung sliding, B-lines, pleural effusion, consolidation, bronchograms, pleural line irregularity, or subpleural consolidations. The abnormal findings (Fig. 1) were defined as: absence of lung sliding- absence of the regular rhyth- mic movement between the visceral and parietal pleura synchro- nized with respirations [17]; B-lines- discrete laser-like vertical hyperechoic reverberation artifacts that arise from the pleural line, erase A-lines, move in synchrony with respirations, and extend to the bottom of the image [17]; pleural effusion- anechoic or hypoe- choic space between the visceral and parietal pleura with variation of the Interpleural distance during respirations [17]; consolidation- subpleural echo-poor or tissue-like region [17]; bronchograms- hyperechoic linear branching structures within regions of consoli- dation [23]; pleural line irregularity- irregular, thickened or fragmented pleural line [17]; subpleural consolidation- subcen- timeter echo-poor pleural-based region of consolidation [12,18,24]. For the purposes of IRR calculations, a single B-line was considered an abnormal finding. Sonographers also recorded if they were aware of clinical information. Each study ultrasound and the associated clips (longitudinal and transverse) from the 12 zones for each patient were independently reviewed by one of 3 expert sonographers within Q-path and an identical data col- lection form was completed. The expert reviewers were blinded to original sonographer’s datasheet as well as any clinical informa- tion. The expert reviewer rated the quality of the entire study on a scale of 1 (no recognizable structures, no objective data can be gathered in one or more zones) to 5 (at least minimal criteria met for a complete lung US study protocol, all structures imaged with excellent Image quality and diagnosis completely supported). Studies with overall quality rated <3 were excluded from our anal- ysis. The length of time to perform each complete study ultrasound was estimated by the timestamps on the first and last saved clips. Our primary outcome measures were IRR estimates calculated for all lung zones individually, based on an overall assessment of normal versus abnormal. Secondary outcome measures were IRR estimates for zone groupings based on anatomic locations, stratifi- cation based on age (<4 versus >=4 years), individual lung ultra- sound abnormalities, and a three-item composite indicator for pneumonia. Stratification by age was determined a priori based on the authors’ prior clinical experience with patient cooperativity during point-of-care lung ultrasound. The composite indicator for pneumonia was based on three commonly-cited ultrasound abnor- malities diagnostic of Bacterial pneumonia: consolidation, focal B- lines, and pleural effusion [4]. For this Composite outcome, any of these three findings within an individual lung zone was considered

a positive, or abnormal, result.

Data analysis

Descriptive statistics (i.e., medians with interquartile ranges and frequencies with proportions for continuous and categorical variables, respectively) of Demographic and clinical factors were calculated.

To calculate interrater reliability (IRR), we created a dataset in which each patient was represented by 24 observations (one

Fig. 1. Lung artifacts and abnormal findings. A, B-lines; B, pleural effusion; C, consolidation; D, subpleural consolidation; E, pleural line irregularity; F, bronchograms.

observation per zone per rater), where zones were represented by the twelve measurements per patient and rater was defined as study sonographer versus expert reviewer. Each observation was categorized as ”normal” or ”abnormal”, based on the sonographer assessment. To account for correlation between measurements from the same patient, we estimated a multilevel mixed-effects linear regression model with the normal/abnormal designation as the dependent variable and patient-level and zone-level identifiers as nested random intercept effects (i.e., zone-level nested within patient-level). As a measure of IRR, we calculated the intraclass correlation (ICC) using the random-effects parameters derived from this model as follows:

ICC = zone-level variance/(zone-level variance

+ patient-level variance + residual (rater-level) variance)

As supplementary measures of IRR, we also calculated raw per- cent agreement and the kappa statistic. Both ICC and kappa values are reported on a scale of 0 to 1 (poor to slight agreement: <0.20; fair agreement: 0.21-0.40; moderate agreement: 0.41-0.60; sub- stantial agreement: 0.61-0.80; and near perfect agreement:

>0.81) [10,25].

The following sample size calculations assumed a two-sided 95% confidence interval (CI), a target ICC of at least moderate reli- ability (0.50), and 24 observations per patient. We set our target level of precision for estimated CIs to be +-10%. We required 50 sub- jects to estimate a CI with a precision of +-10% and 75 subjects to achieve a precision of +-8.5%. For analyses of specific subsets of zones, sample sizes of 50 and 75 patients would provide precision levels of +-11% and +- 9%, respectively. Thus, we set our target enroll- ment at 75 patients.

Results

A total of 105 patients were enrolled (Fig. 2). Twenty-three were later excluded after enrollment for one or more of the follow- ing reasons: no radiograph was obtained, the sonographer had

Original Enrollment Sample n=105

Withdrawn prior to ultrasound n=11

Additional exclusions n=23

(Image quality <3, n=3) (No CXR performed, n=7)

(Sonographer aware of CXR, n=13)

Final Sample for ICC analysis n=71

CXR, chest radiograph; ICC, intraclass correlation

Fig. 2. Patient enrollment flowchart. CXR, chest radiograph; ICC, intraclass correlation.

knowledge of the chest radiograph result, or ultrasound image quality was poor. An additional eleven patients were withdrawn after enrollment for the following reasons: discharged before the ultrasound could be performed (4 patients), unable to tolerate the full study ultrasound (3 patients), incomplete data from the

treating physician (3 patients), and parental concern for skin reac- tion to the ultrasound gel (1 patient). A final sample of 71 patients,

Table 2

Interrater reliability of lung ultrasound in 71 pediatric patients.

with a total of 852 lung zones imaged, was included for analysis of interrater reliability. Patient demographics and characteristics are outlined in Table 1.

Lung zones Percent

agreement

Kappa Intraclass correlation (ICC), 95%

confidence interval

Of the 71 patients included for analysis, 46 (65%) had the study

All zones

85.5%

0.56

0.46 (0.41, 0.52)

ultrasound performed by an expert sonographer. For 13 patients

All zones, age < 4 (n = 29)

85.9%

0.56

0.49 (0.41, 0.58)

(18%), the sonographer was aware of clinical information (but

All zones, age >= 4 (n = 42)

85.3%

0.56

0.45 (0.37, 0.52)

not the radiograph result) pertaining to the patient’s presentation. Technical difficulties reported by the sonographers included patient cooperativity in 15 (21%) patients and body habitus in 7

Anatomic regions

Right-sided zones

85.8%

0.56

0.43 (0.35, 0.51)

Left-sided zones

85.2%

0.56

0.36 (0.28, 0.44)

Anterior zones (bilateral)

88.2%

0.53

0.40 (0.29, 0.51)

(10%) patients. Image quality, as judged by the expert reviewer,

Anterior, right

87.9%

0.55

0.42 (0.23, 0.62)

was 3 out of 5 in 24 (34%) patients, 4 out of 5 in 41 (58%) patients,

Anterior, left

88.6%

0.51

0.23 (0.06, 0.40)

and 5 out of 5 in 6 (8%) patients. The median length of time (IQR) to perform the study ultrasound was 9 min [7,11].

The sonographer assessment of normal versus abnormal, when calculated across each of the 852 zones, demonstrated moderate agreement with an ICC of 0.46 (95% CI: 0.41, 0.52), with a percent agreement of 86%, and a kappa of 0.56. Similar reliability character- istics with moderate agreement were found when the patient pop- ulation was stratified by age (<4 vs. >=4 years). ICC values for all zones and for specific anatomic regions are presented in Table 2. Right-sided zones had moderate agreement while left-sided zones showed only fair agreement. Anterior, lateral and posterior zones

Lateral zones (bilateral)

82.8%

0.52

0.42 (0.30, 0.53)

Posterior zones (bilateral)

85.5%

0.61

0.49 (0.38, 0.60)

Superior zones

84.6%

0.51

0.40 (0.32, 0.49)

Inferior zones

86.4%

0.61

0.50 (0.42, 0.58)

All ICC estimates derived from multilevel mixed-effects models with lung zones nested within patient.

Table 3

Interrater reliability of lung ultrasound for specific abnormalities in 71 pediatric patients.

demonstrated moderate agreement, although the left-sided ante- rior zones specifically showed only fair agreement. Inferior and superior zones also revealed moderate agreement. When ICC was stratified by the study sonographer experience for all lung zones, the ICC for expert sonographers was moderate (0.57 (95% CI:

0.51, 0.64)) and for intermediate sonographers was fair (0.34

B-lines

91 (11)

92.3%

0.59

0.52 (0.46, 0.57)

(95% CI: 0.24, 0.44)). When stratified by image quality, the ICC

Pleural effusion

5 (0.6)

99.4%

0.44

0.40 (0.34, 0.45)

for studies with an image quality of 3 was fair (0.35 (95% CI: 0.26, 0.46)) and for combined image qualities of 4 and 5 was mod-

erate (0.53 (95% CI: 0.47, 0.60)).

Consolidation

Bronchograms

Pleural line irregularity Subpleural

27 (3)

0 (0)

59 (7)

41 (5)

96.4%

99.4%

88.2%

93.4%

0.46

n/ab 0.23

0.34

0.39 (0.33, 0.44)

n/ab

0.16 (0.10, 0.23)

0.31 (0.25, 0.37)

When assessing specific lung ultrasound abnormalities, B-lines,

consolidation

pleural line irregularity, subpleural consolidations and lung consol-

Composite for

91 (11)

90.4%

0.58

0.52 (0.46, 0.57)

pneumoniac

Abnormality Abnormal zonesa

Percent agreement

Kappa Intraclass

correlation (ICC),

95% confidence interval

Table 1

Demographic characteristics of patients in final sample for interrater reliability analysis.

All ICC estimates derived from multilevel mixed-effects models with lung zones nested within patient.

a Values represent frequency (proportion) of zones rated by the expert ultra- sound review for each given abnormality. Each proportion is based on 852 zones

(i.e., total 12 zones across each of 71 patients).

Demographic characteristics Patients, n = 71

Age (years), median (IQR)

5.6 (2.5-11)

Age < 4

29 (41)

Age >= 4

42 (59)

Sex (male)

43 (61)

Presenting symptoms

Cough 60 (85)

Fever 52 (73)

labored breathing 23 (32)

Wheezing 10 (14)

Chest pain 8 (11)

History of asthma/reactive airway disease 20 (28) Exam findings

Focal rales/crackles 21 (30)

Wheeze 13 (18)

Decreased breath sounds 20 (28)

Chest radiograph findings

Negative for pneumonia 38 (54)

Equivocal for pneumonia 10 (14)

Definite pneumonia 13 (18)

Pleural effusion 2 (3)

Peribronchial cuffing 20 (28)

Disposition from the ED (admitted) 16 (23) Discharge diagnosis

Pneumonia 28 (39)

Bronchiolitis 6 (8)

Asthma/reactive airway disease 8 (11)

URI/viral syndrome/cough 37 (52)

Data are shown as n (%) unless otherwise specified.

ED, emergency department; IQR, interquartile range; URI, upper respiratory tract infection.

b Model could not be estimated due to sparse data.

c Composite indicator for pneumonia combined any of the following findings in an individual zone (focal B-lines, pleural effusion, or consolidation).

idation were the most frequently encountered (Table 3). No patients had absent lung sliding. Intraclass correlation varied from a high of 0.52 (95% CI: 0.46, 0.57) for B-lines to a low of 0.16 (95% CI: 0.10, 0.23) for pleural line irregularity. Using a 3-item compos- ite indicator for pneumonia (focal B-lines, pleural effusion, and/or consolidation), the kappa was 0.58 and the ICC was 0.52 (95% CI: 0.46, 0.57).

Discussion

The accuracy and reliability of the most commonly utilized diagnostic methods for assessing pediatric patients with respira- tory complaints and concern for pneumonia (i.e., auscultation and chest radiography) have been shown to have practical limita- tions [10,11,26-29]. The interrater reliability of specific examina- tion findings, including auscultation, ranges from poor to moderate; in one recent study, no individual examination finding had substantial agreement and only three findings (wheeze, retrac- tions, and respiratory rate) had acceptable agreement [10]. Chest radiograph interpretation also has significant inter-observer and intra-observer variability with reliability ranges reported from minimal to moderate depending on the specific findings assessed, diagnoses evaluated, interpreting provider specialty and the

geographical setting [26-29]. As lung POCUS has several advan- tages over chest radiography, namely lack of radiation, perfor- mance at the bedside with immediate results, and availability in resource-limited settings, it is important to more clearly understand the reliability of lung POCUS findings given recent meta-analyses supporting its potential value for the diagnosis of pediatric CAP [3,4].

Prior published assessments of interrater reliability of lung POCUS have been limited to specific adult patient sub-populations [30], specific artifacts (e.g. B-lines) [31], review by non-emergency providers [26,28,32], or have largely focused on only a limited num- ber of findings (consolidation, effusion or interstitial disease) [18] or the overall interpretation and diagnosis [7,21,32]. We chose to investigate the interrater reliability for specific lung zones, anatomic groupings, and specific POCUS findings among pediatric patients being evaluated for pneumonia, as this knowledge is essential before utilizing ultrasound diagnostically.

In a pediatric population with acute respiratory complaints and Clinical concern for pneumonia, we observed moderate interrater reliability when assessing for any POCUS abnormality but variable interrater reliability for specific findings. The median length of time to perform the study ultrasound, 9 min, was within the previ- ously reported range of 5.5-10 min [8,18,28,32-34]. Right-sided zones demonstrated moderate IRR while left-sided zones, specifi- cally left-sided anterior zones, showed only fair agreement. This may imply that structures visualized within the left anterior lung zones of the pediatric thorax, such as the cardiac silhouette and thymus, make these zones less reliable for assessing lung POCUS findings. These findings may also suggest that lung POCUS training might place additional emphasis on these more difficult zones.

We did not see a significant difference between groups when stratified by age (<4 years versus >=4 years) with both groups showing moderate agreement. A previous study of physical exam- ination findings in pediatric patients with suspected pneumonia suggested that age may play a role in agreement due to differences in size and pathophysiology [10]. Our results do not support this prior conclusion. Furthermore, although we suspected that younger patients might have poor cooperativity, leading to decreased agreement, our results did not show this. Our results do suggest that higher image quality is associated with improved IRR as the ICC for studies with an image quality of 3 was fair com- pared to moderate agreement for combined image qualities of 4 and 5. Therefore, factors related to image quality (including sono- grapher, patient, and probe selection considerations) may prove to be important in improving reliability.

We found a difference in reliability when lung POCUS was per- formed by expert sonographers versus intermediate sonographers, with moderate agreement for the experts and fair agreement for the less experienced, although not novice, group. Prior literature, focusing on the Interobserver agreement in the assessment of B- lines only, showed that expert/novice pairs performed similarly to expert/expert pairs when evaluating that single finding on lung POCUS [31]. The difference in our findings may be due to several factors. B-lines were shown to be the most reliable of the individ- ual findings in this study, with the highest ICC. This suggests that B-lines, either by their appearance or by way of usual POCUS train- ing programs, may be the most recognizable or least-operator dependent individual finding. Our inclusion of the multiple other POCUS lung findings, that are possibly more operator-dependent or less commonly encountered, may account for the differences observed when stratified by experience.

When evaluating additional lung POCUS abnormalities, we

found the other findings (pleural effusion, consolidation, pleural line irregularity, and subpleural consolidations) to have poor to fair agreement, with pleural line irregularity having the lowest reliabil- ity. These findings were also much less encountered in our patient population when compared to B-lines (0.6%-7% of zones versus 11% of zones, respectively). This may have implications for training and teaching of lung POCUS findings in the future, suggesting that

specific findings may need more attention or that more experience may be necessary to reliably identify all lung POCUS findings. As the clinical significance of some of these findings has not yet been clearly defined, first improving our reliability may afford us the opportunity to better define their significance.

When we assessed a composite indicator for pneumonia, including the most common POCUS Diagnostic findings for CAP (consolidation, focal B-lines, and pleural effusion), we found relia- bility to be in the moderate range with an ICC of 0.52. As the fre- quency of B-lines was greater than that of either other abnormality (B-lines 11%, pleural effusion 0.6%, consolidation 3%) this composite may be weighted heavily towards the reliability of the assessment of B-lines. Importantly, the reliability of consol- idation as an individual finding, the most commonly cited of the diagnostic findings, was only fair with an ICC of 0.39. This value is lower than the previously published kappa range of 0.55-0.93 for lung consolidation [18,26,28]. The variability from the prior lit- erature may reflect the differences in prevalence of consolidation seen in the study populations or differing methodology with vari- able numbers of sonographers and expert reviewers in each study. This variability may imply that larger, multi-center studies to assess IRR across clinical settings and providers could have benefit in providing more accurate assessment of reliability.

Limitations

This study has limitations that must be considered when inter- preting the results. First, we evaluated a specific patient population of pediatric patients with acute respiratory complaints and possi- ble pneumonia. All study patients underwent chest radiography given the degree of clinical concern for pneumonia. While this lim- its our results to a specific subpopulation of pediatric patients pre- senting to the emergency department, we believe this is a relevant population as these children would be ideal for utilizing lung POCUS to assist in diagnosis of CAP pneumonia. Second, this anal- ysis was restricted to a single tertiary care pediatric ED, where multiple providers have completed dedicated training in POCUS, which limits the generalizability of the findings. In addition, the methodology of this study limits the interpretation of the results. A linear high frequency probe was utilized for all study patients, regardless of age or body habitus, as consistent with prior pediatric lung POCUS literature [18,20,35]. A default depth of 5 cm was cho- sen to diminish the disadvantages of this probe type in distinguish- ing B-lines from Z-lines, most frequently referenced to fade at a depth of 2-4 cm [36-38]. We excluded patients with overall image quality of <3, which may have affected our IRR estimates, however these low quality studies would not be considered adequate for diagnosis or medical decision making at our institution. For our IRR estimates, we compared the assessment of lung ultrasound findings as assessed in real time by the primary sonographer to the subsequent assessment of recorded video clips by an expert reviewer. As this was not a direct comparison, we cannot assess the reliability of the acquisition of lung ultrasound findings, only the reliability of the interpretation of the video recordings. The expert reviewer, although blinded to the initial interpretation and clinical details, reviewed all clips from a single patient in one sitting (not in a randomized fashion) so findings visualized in one zone may have affected the interpretation of other zones. For this reason, we utilized the ICC estimates for reliability to account for nesting which may have occurred within each patient. Lastly, this study focused on anatomic groupings and individual abnormal findings but did not assess the overall interpretation of the lung ultrasound for the diagnosis of pneumonia.

Conclusion

We found moderate interrater reliability of lung POCUS for the assessment of pediatric patients with concern for pneumonia.

For diagnosing pneumonia, a composite indicator of typical ultra- sound findings had moderate reliability with B-lines having the highest individual reliability. Right-sided zones demonstrated moderate agreement while left-sided zones, specifically left-sided anterior zones, showed only fair agreement. Further assessment of lung POCUS reliability may be necessary before implementing lung POCUS to evaluate patients with suspected pneumonia. Larger studies across clinically heterogeneous Pediatric populations would inform scanning techniques and refine standardized inter- pretation of specific findings.

Funding

This work was supported by the Dr. Michael Shannon Emer- gency Medicine Award (Boston Children’s Hospital).

Author contributions

Study concept and design (CG, MM, JL, AM, RV, RB), acquisition of the data (CG, JL, AM, RV, RB), analysis and interpretation of the data (CG, MM, RB), drafting of the manuscript (CG, MM, RB), critical revision of the manuscript for important intellectual content (CG, MM, JL, AM, RV, RB), statistical expertise (MM), and acquisition of funding (CG).

Declarations of interest

CG, MM, JL, AM, RV, and RB have no competing interests to report.

References

  1. Chavez MA, Shams N, Ellington LE, Naithani N, Gilman RH, Steinhoff MC, et al. Lung ultrasound for the diagnosis of pneumonia in adults: a systematic review and meta-analysis. Respir Res 2014;15:50.
  2. Ye X, Xiao H, Chen B, Zhang S. Accuracy of lung ultrasonography versus chest radiography for the diagnosis of adult community-acquired pneumonia: review of the literature and meta-analysis. PLoS One 2015;10(6):e0130066.
  3. Pereda MA, Chavez MA, Hooper-Miele CC, Gilman RH, Steinhoff MC, Ellington LE, et al. Lung ultrasound for the diagnosis of pneumonia in children: a meta- analysis. Pediatrics. 2015;135(4):714-22.
  4. Balk DS, Lee C, Schafer J, Welwarth J, Hardin J, Novack V, et al. Lung ultrasound compared to chest X-ray for diagnosis of pediatric pneumonia: a meta- analysis. Pediatr Pulmonol 2018;53(8):1130-9.
  5. Boursiani C, Tsolia M, Koumanidou C, Malagari A, Vakaki M, Karapostolakis G, et al. Lung ultrasound as first-line examination for the diagnosis of community-acquired pneumonia in children. Pediatr Emerg Care 2017;33 (1):62-6.
  6. Guerra M, Crichiutti G, Pecile P, Romanello C, Busolini E, Valent F, et al. ultrasound detection of pneumonia in febrile children with respiratory distress: a prospective study. Eur J Pediatr 2016;175(2):163-70.
  7. Jones BP, Tay ET, Elikashvili I, Sanders JE, Paul AZ, Nelson BP, et al. Feasibility and safety of substituting lung ultrasonography for chest radiography when diagnosing pneumonia in children: a randomized controlled trial. Chest. 2016;150(1):131-8.
  8. Stadler JAM, Andronikou S, Zar HJ. Lung ultrasound for the diagnosis of community-acquired pneumonia in children. Pediatr Radiol 2017;47 (11):1412-9.
  9. Yadav KK, Awasthi S, Parihar A. Lung ultrasound is comparable with chest roentgenogram for diagnosis of community-acquired pneumonia in hospitalised children. Indian J Pediatr 2017;84(7):499-504.
  10. Florin TA, Ambroggio L, Brokamp C, Rattan MS, Crotty EJ, Kachelmeyer A, et al. Reliability of examination findings in suspected community-acquired pneumonia. Pediatrics 2017;140(3).
  11. Shah SN, Bachur RG, Simel DL, Neuman MI. Does this child have pneumonia? The rational clinical examination systematic review. JAMA. 2017;318 (5):462-71.
  12. Basile V, Di Mauro A, Scalini E, Comes P, Lofu I, Mostert M, et al. Lung ultrasound: a useful tool in diagnosis and management of bronchiolitis. BMC Pediatr 2015;15:63.
  13. Caiulo VA, Gargani L, Caiulo S, Fisicaro A, Moramarco F, Latini G, et al. Lung ultrasound in bronchiolitis: comparison with chest X-ray. Eur J Pediatr 2011;170(11):1427-33.
  14. Cohen JS, Hughes N, Tat S, Chamberlain JM, Teach SJ, Boniface K. The utility of Bedside lung ultrasound findings in bronchiolitis. Pediatr Emerg Care 2017;33 (2):97-100.
  15. Varshney T, Mok E, Shapiro AJ, Li P, Dubrovsky AS. Point-of-care lung ultrasound in young children with Respiratory tract infections and wheeze. Emerg Med J 2016;33(9):603-10.
  16. Dankoff S, Li P, Shapiro AJ, Varshney T, Dubrovsky AS. Point of care lung ultrasound of children with acute asthma exacerbations in the pediatric ED. Am J Emerg Med 2017;35(4):615-22.
  17. Volpicelli G, Elbarbary M, Blaivas M, Lichtenstein DA, Mathis G, Kirkpatrick AW, et al. International evidence-based recommendations for point-of-care lung ultrasound. Intensive Care Med 2012;38(4):577-91.
  18. Shah VP, Tunik MG, Tsung JW. Prospective evaluation of point-of-care ultrasonography for the diagnosis of pneumonia in children and young adults. JAMA Pediatr 2013;167(2):119-25.
  19. Iorio G, Capasso M, Prisco S, De Luca G, Mancusi C, Lagana B, et al. Lung ultrasound findings undetectable by chest radiography in children with community-acquired pneumonia. Ultrasound Med Biol 2018;44(8):1687-93.
  20. Lissaman C, Kanjanauptom P, Ong C, Tessaro M, Long E, O’Brien A. Prospective observational study of point-of-care ultrasound for diagnosing pneumonia. Arch Dis Child 2019;104(1):12-8.
  21. Tsung JW, Kessler DO, Shah VP. Prospective application of clinician-performed lung ultrasonography during the 2009 H1N1 influenza A pandemic: distinguishing viral from bacterial pneumonia. Crit Ultrasound J 2012;4(1):16.
  22. Copetti R, Cattarossi L. ultrasound diagnosis of pneumonia in children. Radiol Med 2008;113(2):190-8.
  23. Weinberg B, Diakoumakis EE, Kass EG, Seife B, Zvi ZB. The air bronchogram: sonographic demonstration. AJR Am J Roentgenol 1986;147(3):593-5.
  24. Liu XL, Lian R, Tao YK, Gu CD, Zhang GQ. Lung ultrasonography: an effective way to diagnose community-acquired pneumonia. Emerg Med J 2015;32 (6):433-8.
  25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
  26. Ambroggio L, Sucharew H, Rattan MS, O’Hara SM, Babcock DS, Clohessy C, et al. Lung ultrasonography: a viable alternative to chest radiography in children with suspected pneumonia? J Pediatr 2016;176:93-8 [e7].
  27. Elemraid MA, Muller M, Spencer DA, Rushton SP, Gorton R, Thomas MF, et al. Accuracy of the interpretation of chest radiographs for the diagnosis of paediatric pneumonia. PLoS One 2014;9(8):e106051.
  28. Ellington LE, Gilman RH, Chavez MA, Pervaiz F, Marin-Concha J, Compen- Chang P, et al. Lung ultrasound as a diagnostic tool for radiographically- confirmed pneumonia in low resource settings. Respir Med 2017;128:57-64.
  29. Neuman MI, Lee EY, Bixby S, Diperna S, Hellinger J, Markowitz R, et al. Variability in the interpretation of chest radiographs for the diagnosis of pneumonia in children. J Hosp Med 2012;7(4):294-8.
  30. Millington SJ, Arntfield RT, Guo RJ, Koenig S, Kory P, Noble V, et al. Expert agreement in the interpretation of lung Ultrasound studies performed on Mechanically ventilated patients. J Ultrasound Med 2018;37:2659-65.
  31. Gullett J, Donnelly JP, Sinert R, Hosek B, Fuller D, Hill H, et al. Interobserver agreement in the evaluation of B-lines using bedside ultrasound. J Crit Care 2015;30(6):1395-9.
  32. Chavez MA, Naithani N, Gilman RH, Tielsch JM, Khatry S, Ellington LE, et al. Agreement between the World Health Organization algorithm and lung consolidation identified using point-of-care ultrasound for the diagnosis of childhood pneumonia by general practitioners. Lung. 2015;193(4):531-8.
  33. Claes AS, Clapuyt P, Menten R, Michoux N, Dumitriu D. Performance of chest ultrasound in pediatric pneumonia. Eur J Radiol 2017;88:82-7.
  34. Reali F, Sferrazza Papa GF, Carlucci P, Fracasso P, Di Marco F, Mandelli M, et al. Can lung ultrasound replace chest radiography for the diagnosis of pneumonia in hospitalized children? Respiration. 2014;88(2):112-5.
  35. Yilmaz HL, Ozkaya AK, Sari Gokay S, Tolu Kendir O, Senol H. Point-of-care lung ultrasound in children with community acquired pneumonia. Am J Emerg Med 2017;35(7):964-9.
  36. Francisco MJN, Rahal AJ, Vieira FA, Silva PS, Funari MB. Advances in lung ultrasound. Einstein (Sao Paulo) 2016;14(3):443-8.
  37. Lichtenstein DA, Meziere GA, Lagoueyte JF, Biderman P, Goldstein I, Gepner A. A-lines and B-lines: lung ultrasound as a bedside tool for predicting pulmonary artery occlusion pressure in the critically ill. Chest. 2009;136 (4):1014-20.
  38. Yue Lee FC, Jenssen C, Dietrich CF. A common misunderstanding in lung ultrasound: the comet tail artefact. Med Ultrason 2018;20(3):379-84.

Leave a Reply

Your email address will not be published. Required fields are marked *