Introduction   

Randomised controlled trials (RCTs) provide the most reliable evidence for the impacts of medical interventions.1 However, RCTs can be biased by faults in the design, performance, analyses and reporting. Bias is a systemic error that underestimates or overestimates the true effects of an intervention.2 Bias can invalidate the results of RCTs, potentially leading to patients receiving non-beneficial or harmful treatments.3

The Cochrane Collaboration’s tool for assessing the risk of bias in randomised trials was developed to clarify the trial process and increase accuracy.2 The Cochrane Collaboration’s tool has been used to assess the risk of bias in RCTs conducted in various fields, including paediatrics, orthopaedics, urology, neurology and ophthalmology.3–7 Recently, the risk of bias in 20 920 RCTs included in the Cochrane Review was reported.8 In the emergency medicine area, the Cochrane Collaboration’s risk of bias tool was used to evaluate RCTs that assessed simulation-based medical education.9 However, to our knowledge, no study has evaluated the risk of bias in the RCTs referenced in the guidelines for cardiopulmonary resuscitation (CPR). Clinical guidelines are an important tool for knowledge transfer and affect millions of clinicians and patients.10 11 The main advantage of guidelines is that they reduce unjustified variations in patient care, but biased guidelines are potentially harmful and ineffective for the patient.11 Although the Cochrane Collaboration’s tool for assessing the risk of bias was used to conduct an evidence evaluation of RCTs in the 2015 American Heart Association (AHA) guidelines for CPR and emergency cardiovascular care (ECC),12 the assessment of individual items of bias or overall features was not presented.

The purpose of this study was to identify the risk of bias in the RCTs referenced in the 2015 AHA guidelines update for CPR and ECC,12–25 which is used worldwide and considered as the basis for resuscitation.

Methods

We identified the RCTs cited as references in the 2015 AHA guidelines update for CPR and ECC. Articles containing the search term ‘random’ in the title or abstract were extracted. After reviewing the contents of the abstract and text, the studies that were included in the actual randomisation process were confirmed. We included clinical studies in which participants were human patients and excluded animal studies. Studies that analysed existing RCTs and RCTs published in a letter format were also excluded.

We analysed the RCTs using the Cochrane Collaboration’s tool for assessing the risk of bias in randomised trials.1 The following six domains of the Cochrane Collaboration’s tool were selected to evaluate the risk of bias: random sequence generation and allocation concealment for selection bias, blinding of participants and personnel for performance bias, blinding of outcome assessment for detection bias, incomplete outcome data for attrition bias and selective reporting for reporting bias. We did not prespecify other sources of bias that are not identified by the above six domains described because we were not able to simply define the other sources of bias in RCTs of various topics included in the guidelines. The risk of bias for each domain was reported as ‘low’, ‘unclear’, or ‘high’. The criteria for assessing the risk of bias are shown in online supplementary appendix 1. As shown in the study by Zhai et al, real-time randomisation was added to the domain of allocation concealment.5 It was considered low risk when the treatment was assigned at the time of randomisation and allocation concealment was guaranteed. If participants do not know the study objectives, the domain of blinding of participants and personnel was judged as a low risk of bias. In addition, studies with objective/permanent end-points, such as mortality or laboratory data, were evaluated as having a low risk of bias for the blinding of outcome assessment. Two independent reviewers (YC and CK) scored the RCT articles in each domain, and a third reviewer (BK) resolved the discrepancies. Kappa values for the inter-rater agreement between the two reviewers were calculated for each of the six domains.

We identified the number of RCTs in 5-year intervals. The percentages of RCTs with high, unclear and low risks of bias were determined for each of the six Cochrane domains. We examined the risk of bias in RCTs by grouping them into seven topics (basic life support [BLS], adult advanced cardiovascular life support [ACLS], postcardiac arrest care, acute coronary syndrome, neonatal and paediatric resuscitation, education, and others). We also investigated the risk of bias in RCTs based on the type of intervention (drug trial or non-drug trial). The journal’s impact factor (IF) in 2017 and the Journal Citation Report (JCR) categories were identified. We designated journals without an IF as having an IF of zero. The risk of bias in the six domains was identified in the following journal IF groups: IF<5 (low IF), 5≤IF<10 (intermediate IF) and IF≥10 (high IF). The risk of bias for each Cochrane domain was evaluated in RCTs divided into three periods (≤1995, 1996–2009 and ≥2010) based on the year of introduction and the update of the Consolidated Standards of Reporting Trials (CONSORT) statement. We used R V.3.4.0 (www.R-project.org) to plot the numbers and proportions of RCTs over 5-year intervals. Microsoft Excel was used for data collection and creating the graphs showing the risk of bias.

Results

Three hundred RCTs referenced in the 2015 AHA guidelines were identified. After the exclusion of 27 articles, 273 RCTs were selected for analyses. Studies were excluded because they were animal studies (n=23), studies that analysed existing RCTs (n=3), and RCTs published in a letter format (n=1) (online supplementary appendices 2, 3). The RCT articles included in this study were published from 1980 to 2015. The number of RCTs has increased during this period (figure 1), and 90.5% (247/273) of RCTs were published after 1996. A total of 42.1% (115/273) of the RCTs included in this study were published after 2010.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Number of randomised controlled trials (RCTs) referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care stratified into 5-year intervals.

Table 1 shows the baseline characteristics of the RCTs. The median number (IQR) of participants randomised and the number of participants analysed were 140 (59–372) and 130 (54–335), respectively. Education (33.0%, 90/273) was the most common topic cited in the guidelines, followed by neonatal and paediatric resuscitation (18.3%, 50/273) and acute coronary syndrome (14.7%, 40/273). The most common JCR category was critical care medicine (27.5%, 75/273), followed by medicine, general and internal (22.0%, 60/273).

The median IF in 2017 was 5.9 (IQR: 3.7–16.8) (table 2). Seven RCTs were published in journals without an IF. Approximately one-third of the RCTs (95/273, 34.8%) included in the present study were published in journals with 5≤IF<10, and 28.6% (78/273) were published in journals with an IF≥10. The New England Journal of Medicine had the highest IF and was the most cited journal among the RCTs included in this study. Among the included topics, the highest median IF was observed for trials investigating acute coronary syndrome (18.9, IQR: 5.2–53.3) and the lowest was observed for studies investigating neonatal resuscitation (3.7, IQR: 2.3–5.5). Based on the type of intervention, the median IF of drug trials (18.9, IQR: 4.7–79.3) was much higher than that of non-drug trials (5.9, IQR: 3.6–10.1).

The kappa values for inter-rater agreement of the individual domains of the Cochrane Collaboration’s tool were 0.86 for random sequence generation, 0.89 for allocation concealment, 0.89 for the blinding of the participants and personnel, 0.84 for the blinding of outcome assessment, 0.76 for incomplete outcome data and 0.71 for selective reporting.

The proportion of trials with each risk of bias rating for the six domains is shown in figure 2. A large proportion of trials displayed a high risk of bias (inadequate method) for the domain of blinding of participants and personnel (78.8%, 215/273). Meanwhile, the proportions of trials with inadequate methods (high risk of bias) for the remaining five domains (random sequence generation, allocation concealment, blinding of outcome assessment, incomplete outcome data and selective reporting) were low (≤12.1%). However, of these five domains, the proportions of trials with an unclear risk of bias (poor reporting) for two domains (random sequence generation and allocation concealment) were greater than 30% (38.5%, 105/273% and 34.1%, 93/273, respectively) (figure 2). In the subgroup analysis based on the topics of the trials, ACLS trials had the highest proportion of trials with a low risk of bias for the blinding of participants and personnel (71.9%, 23/32) (figure 3). Larger proportions of trials examining other topics (BLS, postcardiac arrest care, acute coronary syndrome, neonatal and paediatric resuscitation, education, and others) had a high risk of bias for the blinding of participants and personnel (94.7%, 85.7%, 85.0%, 86.0%, 88.9% and 66.7%, respectively) than ACLS trials (25%). ACLS trials represented the highest proportions of trials with an unclear risk of bias for random sequence generation (50%, 16/32) and incomplete outcome data (31.3%, 10/32). Education trials represented the highest proportion of trials with an unclear risk of bias for allocation concealment (58.9%, 53/90).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Risk of bias in randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care based on the six domains of the Cochrane Collaboration’s tool.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Risk of bias in randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care for each of the six domains of the Cochrane Collaboration’s tool stratified by topic. ACLS, adult advanced cardiovascular life support; BLS, basic life support.

Regarding the type of intervention, the proportion of drug trials with a high risk of bias for the blinding of participants and personnel (21.5%, 8/37) was less than that of non-drug trials (87.7%, 207/236) (figure 4). Seventy-five per cent of ACSL trials (24/32) were drug trials, among which 91.7% of drug trials on the ACLS topic (22/24) displayed a low risk of bias for the blinding of participants and personnel.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Risk of bias in randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care for each of the six domains of the Cochrane Collaboration’s tool stratified by the type of intervention.

After stratification according to the three groups of IFs in 2017 [low group (IF<5), intermediate group (5–10) and high group (≥10)], the proportion of trials with a low risk of bias showed an increasing trend as the IF increased, although little difference was observed between the two higher IF groups (intermediate vs high IF group) in the domains of random sequence generation and selective reporting (figure 5). However, the proportions of trials with an unclear risk of bias were still high for the domain of random sequence generation and allocation concealment, even in journals with high IFs (38.5% and 23.1%, respectively). Even after stratification according to the IFs of the year of publication, the proportions of trials with an unclear risk of bias in journals with high IFs were high for the domain of random sequence generation (35.6%) and allocation concealment (22.0%) (online supplementary appendix 4).

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Risk of bias in randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care for each of the six domains of the Cochrane Collaboration’s tool stratified by impact factor (IF).

After stratification by the year of introduction (1996) and update of the CONSORT statement (2010), the proportion of trials with a low risk of bias also tended to increase after the introduction of the statement in 1996 and after its update in 2010 (figure 6) for four domains (random sequence generation, blinding of outcome assessment, incomplete outcome data and selective reporting). Unclear risk of bias proportions was 65.4% for random sequence generation and 57.7% for allocation concealment before the introduction of CONSORT but decreased to 31.3% and 32.2% after the 2010 CONSORT update, respectively. For allocation concealment, the proportion of trials with a low risk of bias also increased after the introduction of the CONSORT statement in 1996 (30.8%–57.6%); however, the proportion did not increase after the update of the CONSORT statement in 2010 (57.6%–54.8%). For blinding of participants and personnel, the proportion of trials with a low risk of bias showed an opposite trend after the introduction of the CONSORT statement in 1996 and after its update in 2010. Drug-trials were 30.8% (8/26) prior to or in 1995, 14.4% (19/132) from 1996 to 2009, and 8.7% (10/115) in in or after 2010.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

Risk of bias in randomised controlled trials referenced in the 2015 American Heart Association guidelines update for cardiopulmonary resuscitation and emergency cardiovascular care for each of the six domains of the Cochrane Collaboration’s tool divided into three periods (≤1995, 1996–2009 and ≥2010) based on the year of introduction and the update of the Consolidated Standards of Reporting Trials statement. 

Discussion

In this study, we assessed the risk of bias in 273 RCTs that were referenced in the 2015 update of the AHA guidelines for CPR and ECC. The largest proportion of trials with a high risk of bias (inadequate methods) was observed for the blinding of participants and personnel (78.8%). This finding might be explained by the fact that only 13.6% (37/273) of the RCTs in this guideline were drug trials that can be sufficiently blinded by placebos. The remaining 86.4% of the RCTs were non-drug trials that did not use placebos. In general, the masking of participants and personnel is difficult in non-drug trials where placebos are not available. In fact, the proportion of drug trials with a high risk of bias for the blinding of participants was less (21.5%) than that in non-drug trials (87.7%) in this study. Furthermore, in some cases, blinding is not performed because of an ethical issue, depending on the outcome of cardiac arrest.

In domains other than the blinding of participants and personnel, the proportion of trials with a high risk of bias was less than 12.1%. However, greater than one-third of trials exhibited unclear risks of bias for random sequence generation (38.5%) and allocation concealment (34.1%), indicating that many trials did not sufficiently report the risk of bias.

Random sequence generation minimises selection bias and balances baseline characteristics between the study and control groups. Methods such as a computerised random number generator, random number table, or coin tossing should be used to ensure a low risk of bias. Sequences generated by an odd or even date of birth or a non-random process using hospital registration numbers will lead to bias. In this study, 38.5% of RCTs showed unclear risk of bias in random sequence generation, and half of the ACLS trials poorly described random sequence generation, possibly because patients experiencing cardiac arrest were not able to be randomly divided using the above methods to minimise the risk of bias for random sequence generation due to ethical issues.

Random sequence generation alone is not sufficient to prevent selection bias, and allocation concealment must be employed to effectively protect unpredictable randomised sequences.2 Central allocation, sequential numbered opaque sealed envelopes (SNOSE) or minimisation should be used to ensure participants and researchers are not aware of assignments in advance. According to Akl et al,26 SNOSE leads to a risk of bias when the patient or participants opens the envelope in advance. The authors recommended the use of simple randomization when the sample size is large. Block randomisation is often used if stratification is required; however, a small block size should be avoided because the allocation can be predicted in advance. Allocation by minimisation is much more difficult to predict than block randomisation because minimisation is a dynamic mode of random allocation that depends on the characteristics of the participants who are already enrolled.26 In this study, 12.1% of RCTs had a high risk of bias for allocation concealment. A total of 34.1% of trials showed an unclear risk of bias for allocation concealment, which is less than the percentages reported by a recent study analysing the 20 920 RCTs included in the Cochrane reviews (57.5%)8 or RCTs of simulation-based interventions in emergency medicine (48.5%).9 However, in a paper analysing the RCTs published in four general medical journals (The BMJ, JAMA, The Lancet and The NEJM), 26% of trials published in 200227 and 22% of trials published in 201526 poorly described information regarding allocation concealment. This finding is similar to our study, as the proportion of RCTs with an unclear risk of bias was 23.1% in the group with an IF≥10. A potential explanation is that journals with high IFs have been included in the 2015 AHA guidelines (28.6% of trials included in this study had an IF greater than 10).

If outcome assessors know the assignment of the intervention, detection bias will occur in this assessment.2 The lack of blinding of outcome assessors can lead to different outcomes, and clinicians must decide who will assess the outcome of RCTs at the planning stage. In this study, the proportions of trials with unclear and high risks of bias for the blinding of outcome assessment were relatively low (8.4% and 2.2%, respectively) compared with other domains because we assessed that domain as a low risk when the study end-point was objective or permanent, such as mortality or laboratory findings, which were unlikely to affect the bias even if the outcome assessors were not blinded. Online supplementary appendix 5 shows the end-point of the trials included in this study. Many of the trials were objective (eg, the presence or absence of return of spontaneous circulation, survival or death, admission or discharge and the results of laboratory findings) and, thus, are unlikely to affect the outcome assessment.

We determined that 22.3% of the RCTs showed an unclear risk of bias for incomplete outcome data, which suggests insufficient reporting on attrition and exclusion. Of the RCTs published in the five top general medical journals in 2005–2007, 13% did not discuss loss to follow-up.26 Additionally, 24.7% of the RCTs included in the Cochrane Review in 2011–2014 showed an unclear risk of bias in the domain of incomplete outcome data.8 The number of participants in each intervention group and reasons for attrition and exclusion should be reported to assess this domain. Studies in which participants are excluded by a per-protocol analysis are considered high risk, while an intention-to-treat analysis is often recommended as the least biased method.2

Dechartres et al 8 reported that the proportions of RCTs with unclear risks of bias in sequence generation, allocation concealment, blinding and incomplete outcome data have decreased from 1986 to 2014. The proportions of RCTs referenced in the 2015 AHA guidelines with an unclear risk of bias also showed decreasing trends after the introduction of the CONSORT statement in 1996 and after its update in 2010 for four domains (random sequence generation, blinding of outcome assessment, incomplete outcome data and selective reporting). However, for random sequence generation and allocation concealment, a larger proportion of RCTs showed an unclear risk of bias than the other four domains in both periods: 1996–2009 and ≥2010. A reporting guideline, such as the CONSORT statement, should be applied to improve reporting.27 When the CONSORT guideline is actively applied, the quality of reporting improves.28–30 However, only 38% of biomedical journals (168) with a high IF used the CONSORT guideline in 2014,31 and only 14% of surgical journals mentioned using a reporting CONSORT guideline in 2017.32 Journals should suggest to authors that the items of the CONSORT guideline should be reported at the time of submission, and authors should consider the guideline beginning at the planning stage of the study. The proportions of drug trials were decreased after the introduction of the CONSORT statement in 1996 and after its update in 2010, which might be the reason why the proportion of trials with a low risk of bias for the blinding of participants and personnel showed a declining trend.

Inadequate or unclear sequence generation, allocation concealment and blinding affect treatment outcomes.33–35 Clinicians should consider that these biases exaggerate the treatment effect when interpreting the results of RCTs, particularly trials with subjective outcomes, and when applying these data to the management of patients. Additionally, the effect of bias on RCTs should be considered when designing a systematic review and creating a guideline.

This study has limitations. The risk of bias assessed using the Cochrane Collaboration tool can be subjective. However, two reviewers evaluated the included studies, and a third party resolved the differences. We also established a detailed protocol (online supplementary appendix 1) for evaluation and tried to reduce discrepancies between the reviewers. We calculated kappa values to evaluate inter-rater agreement and observed almost perfect (0.81–1.00) or substantial (0.61–0.80) agreement. The kappa values for the Cochrane tool used in our study were greater than those observed when this tool was used in physical therapy trials36 and similar to those found in quality reporting of the RCTs in five leading neurology journals.5 Second, if we had contacted the authors, some of the trials scored as an unclear risk of bias could have been clarified and changed. According to Jørgensen et al, uncertainty can be resolved by contacting trial authors or trial registers.36 When evaluating selective reporting in this study, protocols were searched if the trial was registered, but we did not contact the authors. Third, although the reviewers were blinded to the journal’s name during classification to minimise the possibility of classification bias due to IF, the reviewers were able to recognise the journals despite our efforts; thus, the reviewers might have been influenced by the journal’s IF. Fourth, we evaluated the journal’s IF in 2017, but this value may not accurately reflect the impact of the journal at the time the article was published. Nevertheless, we relied on the most recent IF (2017) because the journal’s IF has consistently increased during the study period. Therefore, a direct comparison of the IF values from different years is impossible. Journals with the same IF values in 1980 and 2017 have different impacts (an IF of 1.8 in 1980 is quite different from the same value in 2017). Furthermore, a low IF for a journal does not indicate that the journal is of low quality. IFs may not properly reflect the quality of each journal. For example, many journals are located in the top level in specialist categories, although their IFs are less than five.

Conclusions

In the 2015 AHA guidelines for CPR and ECC, the proportions of RCTs with a high risk of bias were high for the blinding of participants and personnel. In ACLS and drug trials, the proportions of trials with a low risk of bias for the blinding of participants and personnel were high. The proportions of RCTs with an unclear risk of bias were high for random sequence generation and allocation concealment.

These proportions were reduced after the introduction of the CONSORT statement but remained at 31.3% for random sequence generation and 33.2% for allocation concealment after the 2010 update of the CONSORT statement.

The risk of bias should be considered when interpreting and applying the CPR guidelines in clinical settings. Journals should make an effort to provide authors with information for assessing the risk of bias and should request the use of a reporting guideline, such as the CONSORT guideline, at the submission stage to minimise the risk of bias. In addition, the authors also should plan their research to reduce the risk of bias and clearly report all domains of the Cochrane Collaboration’s tool.