Systematic reviewers commonly contact study authors but do so with limited rigor

doi:10.1016/j.jclinepi.2008.08.002

Journal of Clinical Epidemiology

Volume 62, Issue 2, February 2009, Pages 138-142

https://doi.org/10.1016/j.jclinepi.2008.08.002 Get rights and content

Abstract

Objectives

Author contact can enhance the quality of systematic reviews. We conducted a systematic review of the practice of author contact in recently published systematic reviews to characterize its prevalence, quality, and results.

Study Design and Setting

Eligible studies were systematic reviews of efficacy published in 2005–2006 in the 25 journals with the highest impact factor publishing systematic reviews in clinical medicine and the Cochrane Library, identified by searching MEDLINE, EMBASE, and the Cochrane Library. Two researchers determined whether and why reviewers contacted authors. To assess the accuracy of the abstracted data, we surveyed reviewers by e-mail.

Results

Forty-six (50%) of the 93 eligible systematic reviews published in top journals and 46 (85%) of the 54 eligible Cochrane reviews reported contacting authors of eligible studies. Requests were made most commonly for missing information: 40 (76%) clinical medicine reviews and 45 (98%) Cochrane reviews. One hundred and nine of 147 (74%) reviewers responded to the survey, and reported a higher rate of author contact than apparent from the published record.

Conclusion

Although common, author contact is not a universal feature of systematic reviews published in top journals and the Cochrane Library. The conduct and reporting of author contact purpose, procedures, and results require improvement.

Section snippets

Study eligibility

Eligible systematic reviews were: (1) reviews of treatment efficacy; (2) published in 2005–2006; (3) published in either (a) one of the 25 journals with the highest impact factor publishing systematic reviews in clinical medicine or (b) the Cochrane database of systematic reviews; (3) reported a systematic and comprehensive search strategy, eligibility criteria, and conducted quality assessment of eligible primary studies; and (4) included at least three randomized controlled trials.

Results

We identified 147 systematic reviews: top clinical medicine journals published 93 and The Cochrane Library published 54 reviews (Fig. 1). Table 1 describes the number of reviews identified in each eligible journal. Most reviews include some mention of author contact, with greater frequency noted in Cochrane reviews (P < 0.001) (Table 2).

The most common reason motivating author contact was seeking incomplete data (n = 76, 52%); very few reviews reported contacting authors to verify the data

Statement of principal findings

Most systematic reviewers describe contacting authors of included studies. Cochrane reviews reported contacting authors more frequently than reviews published in journals. Reviewers contact authors of eligible studies more often than reported in the published record. Both Cochrane and journal systematic reviews infrequently and incompletely report the results of author contact.

Strengths and limitations

Strengths of this review include: a comprehensive search strategy; inclusion of both Cochrane reviews and reviews

Conclusions

Incomplete reporting of study methods and results may negatively impact the quality of systematic reviews, and lead, in the case of reporting and publication bias, to an overestimation of treatment effects. Author contact is an important step in the systematic review methodology, which, when successful, may serve to avoid bias and improve the strength of the inferences that result from the systematic reviews. Further research on this issue, although necessary, requires important improvements in

Acknowledgments

We wish to thank all the reviewers who responded very generously to our survey. No funding was received for this project.

References (8)

P.J. Devereaux et al.
An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods
J Clin Epidemiol
(2004)
D. Moher et al.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses
Lancet
(1999)
A.W. Chan et al.
Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors
BMJ
(2005)
T.A. Furukawa et al.
Association between unreported outcomes and effect size estimates in Cochrane meta-analyses
JAMA
(2007)

There are more references available in the full text version of this article.

Cited by (58)

Reproducibility of Exercise Interventions in Randomized Controlled Trials for the Treatment of Rotator Cuff-Related Shoulder Pain: A Systematic Review
2024, Archives of Physical Medicine and Rehabilitation
To determine the reproducibility of exercise therapy interventions in randomized controlled trials for rotator cuff-related shoulder pain (RCRSP).
Data sources included Medline, Cumulative Index to Nursing and Allied Health Literature, Cochrane Central Register of Controlled Trials, and SPORTDiscus from studies published from database inception to April 23, 2022.
Randomized controlled trials studying the use of exercise therapy for RCRSP.
Two reviewers extracted exercise reporting details from all studies using the Template for Intervention Description and Replication (TIDieR) and the modified Consensus on Exercise Reporting Template (CERT). The same 2 reviewers assessed risk of bias of all studies using Cochrane Risk of Bias Tool version 2.0.
For 104 studies meeting inclusion criteria, the average number of items reported on the TIDieR was 5.27 (SD 2.50, range 1-12 out of 12) and 5.09 (SD 4.01, range 0-16 out of 16) on the CERT. Improved reporting over time was seen on both the TIDieR and CERT dating back to 1993 and through April 23, 2022. When comparing groups of studies published before and after the TIDieR (2014) and CERT (2016) were established, a statistically significant increase in median scores was noted on the TIDieR (P=.02) but not the CERT (P=.31). Quality of exercise therapy reporting was highest in studies with “low risk” of bias, and lowest in studies with “high risk” of bias on the RoB-2.
Overall exercise reporting in trials for RCRSP is incomplete despite the development of the TIDieR and CERT checklists. This has implications for translating evidence into practice.
Systematic reviewers used various approaches to data extraction and expressed several research needs: a survey
2023, Journal of Clinical Epidemiology
Data extraction is a prerequisite for analyzing, summarizing, and interpreting evidence in systematic reviews. Yet guidance is limited, and little is known about current approaches. We surveyed systematic reviewers on their current approaches to data extraction, opinions on methods, and research needs.
We developed a 29-question online survey and distributed it through relevant organizations, social media, and personal networks in 2022. Closed questions were evaluated using descriptive statistics, and open questions were analyzed using content analysis.
162 reviewers participated. Use of adapted (65%) or newly developed extraction forms (62%) was common. Generic forms were rarely used (14%). Spreadsheet software was the most popular extraction tool (83%). Piloting was reported by 74% of respondents and included a variety of approaches. Independent and duplicate extraction was considered the most appropriate approach to data collection (64%). About half of respondents agreed that blank forms and/or raw data should be published. Suggested research gaps were the effects of different methods on error rates (60%) and the use of data extraction support tools (46%).
Systematic reviewers used varying approaches to pilot data extraction. Methods to reduce errors and use of support tools such as (semi-)automation tools are top research gaps.
Author queries via email text elicited high response and took less reviewer time than data forms – a randomised study within a review
2021, Journal of Clinical Epidemiology
Citation Excerpt :
Study authors had been contacted in 234 of 319 (73%) Cochrane intervention reviews published in 2016 [3]. Barriers to effective author contact include a lack of guidance on the conduct and reporting of author contact in systematic reviews (SRs) [3,4]. It may be challenging to find correct contact information [3,5,6] and to get authors to respond [5-9].
To compare two strategies for requesting additional information for systematic reviews (SR) from study authors.
Randomised study within a SR of hospital volume-outcome relationships in total knee arthroplasty. We sent personalized email requests for additional information to study authors as either email text (“Email” group) or attachment with self-developed, personalised data request forms (“Attachment” group). The primary outcome was the response rate, the secondary outcomes were the data completeness rate and the reviewer time invested in author contact.
Of 57 study authors, 29 were randomised to the Email group and 28 to the Attachment group. The response rate was 93% for Email and 75% for Attachment (odds ratio 4.5, 95% confidence interval [0.9–24.0]). Complete data were provided by 55% (Email) vs. 36% (Attachment) of authors (odds ratio 2.2 [0.8–6.4]). The mean reviewer time was shorter in the Email (mean ± standard deviation of 20.2±14.4 minutes/author) than the Attachment group (31.8±14.4 minutes/author) with a mean difference of 11.6 [4.1–19.1] minutes/author.
Personalised email requests elicited high response but only moderate data completeness rates regardless of the method (email text or attachment). Email requests as text took less reviewer time than creating attachments.
Educational programs to teach shared decision making to medical trainees: A systematic review
2020, Patient Education and Counseling
Shared decision-making (SDM) is a process of collaboration between patients and clinicians. An increasing number of educational programs to teach SDM have been developed. We aimed to summarize and evaluate the body of evidence assessing the outcomes of these programs.
We conducted a systematic review of studies that aimed to teach SDM to medical trainees. Reviewers worked independently and in duplicate to select studies, extract data and evaluate the risk of bias.
Eighteen studies were included. Most studies focused on residents/fellows (61 %) and combined a didactic component with a practical experience (50 % used a standardized patient). Overall, participants reported satisfaction with the courses. The effects on knowledge, attitudes/confidence and comfort with SDM were small; no clear improvement on SDM skills was noted. Evaluation of clinical behavior and outcomes was limited (3/18 studies). Studies had moderate risk of bias.
Very low quality evidence suggests that educational programs for teaching SDM to medical trainees are viewed as satisfactory and have a small impact on knowledge and comfort with SDM. Their impact on clinical skills, behaviors and patient outcomes is less clear.
Integration of formal and systematic outcomes evaluation (effects on behavior/clinical practice) should be part of future programs.
Contacting authors by telephone increased response proportions compared with emailing: results of a randomized study
2019, Journal of Clinical Epidemiology
Citation Excerpt :
Publications of primary research studies often do not report enough information about the study methods and results to allow systematic reviewers to assess the evidence in the studies [1,2].
The aim of the study was to compare response proportions and research costs of telephone calling vs. continued emailing nonresponding authors of studies included in a systematic review.
Key features of included studies were poorly reported in a systematic review of diabetes quality improvement interventions. We developed a survey to request additional information from contact authors. After three email contact attempts, only 76 of 279 authors (27%) had completed the survey. In this study, we randomly assigned nonresponding authors to contact by telephone calling vs. continued emailing to compare the effect of these strategies on response proportions and research costs.
We randomized 87 authors to telephone and 89 to email contact. Telephone contact increased survey completion (36.7% vs. 20.2%; adjusted risk difference of 15.6% [95% confidence interval: 2.90%, 28.4%]; adjusted odds ratio 2.26 [95% confidence interval: 1.10, 4.76]) but required more time to deliver (20 vs. 10 hours in total; 14 vs. 7 minutes per randomized author; 26 vs. 4 weeks), and cost more (total intervention cost of $504 Canadian dollars vs. $252 for the telephone and email arm, respectively).
Contacting nonresponding authors of included studies by telephone increased response compared with emailing but required more investigator time and had higher cost.
Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results
2019, Journal of Clinical Epidemiology
Citation Excerpt :
Not replying of authors to reviewers is another barrier to obtaining additional data. A low replying prevalence (around 43%) was scored in our cross-sectional study (Table 2) as well as in earlier research studies [30,46]. A high prevalence of opposite outcomes, that is, 26.9% (35/130), was scored for the same simple closed-ended question in the survey compared with what was reported in the published review.
The objective of the study was to assess the prevalence, the reporting quality, the need, and the consequences of contacting of authors by Cochrane reviewers to obtain additional information for their reviews.
Cross-sectional study and survey on all new Cochrane intervention reviews published between January 1, 2016 and January 31, 2017.
The cross-sectional study found that reviewers had contacted or had tried to contact studies to obtain additional information in 73.4% (234/319) of reviews but reported poorly on the methods, outcomes, and consequences of this procedure. Most eligible studies in the reviews were poorly reported, but few reviewers 21.2% (65/306) reported that they had contacted these studies. The survey showed that risk of bias scores, Grading of Recommendations, Assessment, Development and Evaluation scores, the summary primary or secondary outcomes, and the summary effect size of the primary outcome of the review were changed as a consequence of contacting of authors. Thirty-five of one hundred and thirty (26.9%) reviews scored opposite outcomes for the same question in the cross-sectional study compared with the survey.
Our findings on contacting of authors by Cochrane reviewers showed relevant shortcomings in the current standards and transparency of Cochrane reviews. These shortcomings can compromise the validity and reproducibility of these reviews and affect a wide audience.

View all citing articles on Scopus

View full text

Review ArticleSystematic reviewers commonly contact study authors but do so with limited rigor

Abstract

Objectives

Study Design and Setting

Results

Conclusion

Section snippets

Study eligibility

Results

Statement of principal findings

Strengths and limitations

Conclusions

Acknowledgments

J Clin Epidemiol

Lancet

Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors

BMJ

Association between unreported outcomes and effect size estimates in Cochrane meta-analyses

JAMA

Review Article
Systematic reviewers commonly contact study authors but do so with limited rigor