You are here

Article Text

Article menu

Download PDFPDF

XML
Surgery
Research
Frequency of equivocation in surgical meta-evidence: a review of systematic reviews within IBD literature
  1. John D Delaney1,
  2. John T Holbrook2,
  3. Robert K Dewar1,
  4. Patrick J Laws3,
  5. Alexander F Engel4
  1. 1 Colorectal Surgery, Northern Clinical School, University of Sydney, Sydney, New South Wales, Australia
  2. 2 Royal Prince Alfred Hospital, Camperdown, New South Wales, Australia
  3. 3 Prince of Wales Hospital, Sydney, New South Wales, Australia
  4. 4 Department of Colorectal Surgery, Royal North Shore Hospital, Sydney, New South Wales, Australia
  1. Correspondence to Dr John D Delaney; jdel2642{at}uni.sydney.edu.au

Abstract

Objective To assess the level of equivocation among level 1 evidence in ulcerative colitis and Crohn’s disease and determine whether any predisposing factors are present.

Method MEDLINE, Embase, CINHAL and Cochrane were searched from 2006 to 2017. Papers were scored using AMSTAR and categorised into surgical (S), medical (M) or medical and surgical (MS) groups. The ability of each paper to make a recommendation and conclusiveness in doing so was recorded.

Results 278 papers were assessed. 82% (n=227) could make a recommendation, 18% (n=51) could not. There was a significant difference in ability to provide a recommendation between S and M (P=0.003) but not MS and M (P=0.022) nor S and MS (P=0.79). Where a recommendation was made, S papers were more likely to be tempered than M papers (P=0.014) but not MS papers (P=0.987).

Conclusions Surgical meta-evidence within the inflammatory bowel disease domain is more than twice as likely as medical meta-evidence to be unable to provide a recommendation for clinical practice. Where a recommendation was made, surgical reviews were twice as likely to temper their conclusion.

  • inflammatory bowel disease
  • surgery

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2017-018715

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Large sample of papers, the use of multiple independent reviewers and the validity of AMSTAR as a quality assessment tool.

    • The methods used in search and data-retrieval have been clearly outlined, with explicit inclusion and exclusion criteria.

    • The inability of AMSTAR to discriminate between poor methodological quality of a study and poor reporting quality within the paper (internal validity).

    • There are potential avenues for bias in this paper. The use of inflammatory bowel disease (IBD) as a framework may introduce selection bias, particularly given that surgical intervention typically represents a failure of medical therapy in IBD. The assessment of a paper’s level of equivocation is subjective and open to bias. An author’s bias towards a subject may also contribute to a paper’s self-reported level of equivocation and the reasons for equivocation.

    • The assessment of conclusion is subjective, and subtle changes in language may influence the perceived level of confidence and the rationale for uncertainty.

    Introduction

    Methods of aggregate literature review first emerged in the 17th century, developing in an ad hoc fashion until the modern era.1 In the late 1980s, a need to synthesise and understand the increasing volume of medical research drove the development of more sophisticated and systematic techniques.2 Since then, well-conducted systematic reviews and meta analyses have become the gold standard level of evidence in healthcare.3 Such has been the success of these studies in medicine, the process has branched into disciplines as diverse as economics, the social sciences and environmental management.3–5

    Meta-evidence is derivative in nature and as such is dependent on the validity of its input studies to be able to make useful recommendations for clinical practice. When original high-quality trials are combined, they yield more useful meta-evidence that mixed or low quality studies. Unfortunately, many difficulties have been identified that limit the production of high-quality clinical research in surgery6 when compared with medicine, as surgical interventions are typically complex interventions involving the interaction of many independent variables. This creates significant obstacles to generating robust randomised control trials7–9 on surgical topics, and consequently, evidence-based surgery relies heavily on observational studies.10 Audits of methodological rigour within surgical observational studies have been critical.6 10 11 Meta-evidence created from a lower quality selection of original studies has an unreliable foundation. Additionally, an increasing number of papers are being published that examine methodology with surgical meta-evidence. The results of those studies suggest that, in general, meta-evidence within surgery is of poorer methodological quality.12–14 We therefore have a situation where, despite best efforts, surgical meta-evidence is being created from studies of poorer methodological quality than their medical counterparts, and the systematic reviews and meta-analyses themselves are performed with less rigour.

    The research question of this ‘review-of-reviews’ is: what are the factors that influence the ability meta-evidence to make recommendations for clinical practice? Of particular interest is the effect that intervention has; when compared with medical meta-evidence, do the known challenges of original surgical evidence, combined with the historical methodological inferiority of surgical reviews, produce meta-evidence that is more equivocal within the inflammatory bowel disease (IBD) domain?

    Method

    Literature search

    We completed a thorough literature search across MEDLINE, Embase, CINAHL and the Cochrane Database of Systematic Reviews. In addition to the search terms identified in online supplementary appendix 1, a free search of MEDLINE, Embase and CINAHL was completed using the keywords ‘surgery’, ‘meta-analysis’ and either ‘crohn’s’ or ‘ulcerative colitis’. Validated filters for systematic reviews and meta-analyses, specific to each of the databases, were applied.15

    Supplementary file 1

    Definitions

    Papers to be analysed were systematic reviews or meta-analyses, as defined by the Cochrane Collaboration.16 Ulcerative colitis (UC) and Crohn’s disease (CD) were chosen as the framework for this study as they are relatively common, serious conditions,17 with both medical and surgical therapy options.18 The surgical therapies included were derived from International Classification of Diseases, 9th revision, Clinical Modification (ICD-9-CM) procedure codes, along with expert consultation and review of current surgical literature.18–20 Use of ICD-9-CM codes has been previously validated.21

    Retrieved meta-evidence was categorised into groups based on the type of intervention it assessed. Where a medical therapy was considered exclusively, the paper was included in the M group. Where a surgical therapy only was considered, the paper was included in the S group. Where a medical therapy was considered in the context of a surgical therapy, or vice versa, the paper was included in the MS group.

    Papers were further classified as recommendation (R) or no recommendation (NR) based on whether they could provide a recommendation for clinical practice. Each conclusion was rated as either firm (F) or tempered (T) based on the definitiveness of the language used. The conclusion section of each paper was used to assess recommendation and definitiveness. Papers that were R–F were defined as ones that could make a clinical recommendation (positive or negative) using language that was definite and offered minimal or no caveats for the recommendation. Papers that said definitively that there was no difference between interventions, that is, they could confidently not recommend an intervention, were also classed as R–F. Papers that were R–T were those that made a recommendation for practice but offered significant caveats. NR–T papers were not able to offer a recommendation for practice but suggested a recommendation may be possible in the future based on an emerging trend or sound underlying theory. NR–F papers were completely uncertain and could not make a recommendation nor offer further advice due to lack of evidence. See table 1 for a reference list of definitions.

    View this table:
    Table 1

    Definitions for level of recommendation

    The AMSTAR scoring system was used to assess methodological quality.22 AMSTAR consists of 11 individual scoring criteria and is well established as a valid means of assessing meta-evidence.23 AMSTAR is a ‘checklist’ style tool. A higher total AMSTAR score in a paper indicated a more reliable level of methodology.

    Inclusion criteria

    We included systematic reviews or meta-analyses printed between January 2006 and September 2017, inclusive, which assessed a surgical or medical intervention in adults with CD or UC. Review articles were excluded. Papers regarding other IBDs were excluded. The search was limited to full-length publications.

    Data extraction

    Three reviewers examined abstracts (JDD, RKD and PJL). Full text was obtained where abstracts were unable to provide enough information. Updated reviews were used preferentially. Papers that were deemed suitable for inclusion were placed into one of three groups depending on their interventional focus: S, M or MS. JDD, RKD and PJL scored the methodology of the papers via AMSTAR. Any disagreements were resolved by discussion to arrive at a majority decision. Interobserver agreement was assessed using kappa (κ).

    A paper’s recommendation and level of conclusiveness was recorded by JTH and JDD according to the previously stated definitions. Data on the number of papers per review, number of patients included in each review and the 5-year impact factor of the journal in which the paper was published were also recorded. Impact factor was retrieved from the Journal of Citation Reports.24 For papers that included meta-analyses, the number of trials, number of patients and heterogeneity scores (I2 for each) were also extracted.

    Additionally, financial information for each of the papers was extracted based on their description of funding sources or, where that was not available, the affiliations of the first and last authors. Our categories for sponsorship were corporate, government, academia, or those groups in combination, non-government organisations or unclear. An unclear source of funding was recorded where a paper did not offer a conflict of interest disclosure or where a conflict of interest disclosure was offered but the sponsorship of the paper was not clearly outlined.

    Statistical analysis

    All of the collected data were collated into a Microsoft Excel spreadsheet.25 The means of continuous data were compared via analysis of variance (ANOVA). Categorical data were analysed via χ2 test. In both formats, a two-tailed distribution with an alpha level of 0.05 was used. A multivariate ANOVA (MANOVA) assessment of the continuous data set was also performed. Statistical analysis was performed using SPSS V.24.26

    Results

    We identified 739 meta-evidence papers from our initial search. Three hundred and eighty-nine (389) were excluded based on titles or abstracts or because they were duplicated results. Three hundred and fifty papers were reviewed in full. Seventy-two of these papers were excluded (online supplementary appendix 2) (κ=0.8). The 278 included papers were allocated into one of three categories, depending on their interventional focus: S (n=48), M (n=195) or MS (n=35). Descriptive statistics may be found in table 2. The trial flow diagram representing our inclusion and exclusion process is shown in figure 1. Details of the included papers may be found in online supplementary appendix 3.

    Figure 1
    Figure 1

    PRISMA paper inclusion and exclusion flow diagram. IBD, inflammatory bowel disease; M, medical intervention group; MA, meta-analysis; MS, medical and surgical intervention group; n, number of papers; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; S, surgical intervention group; SR, systematic review.

    View this table:
    Table 2

    Paper characteristics

    Overall, 18% of papers (n=51) were unable to make a clinical recommendation based on the available evidence. Within the S group, NR papers made up 31% (n=15). Within MS, NR papers comprised 29% (n=10). Within M, NR papers made up 13% (n=26). A χ2 test was performed, and a significant relationship was found between the intervention type and the likelihood of a paper to be able to make a recommendation (χ2 (2, n=278)=11.049, P=0.004). Comparison of individual groups using χ2 with a Bonferroni correction (α=0.017) revealed a significant difference between S and M (P=0.003) but not between S and MS (P=0.79) nor M and MS (P=0.022).

    One-way ANOVA showed significant differences between S, M and MS groups when comparing the total number of patients (P=0.02) and heterogeneity via I2 (P=0.008). No difference was found in total number of papers, impact factor of journal or AMSTAR rating. Planned contrasts found S papers to have a significantly higher number of patients per review than M papers or MS papers (P=0.001, P=0.009). Contrasts also showed significantly higher heterogeneity via I2 in S when compared with M (P=0.002) and in S and MS combined when compared with M (P=0.016).

    Comparison of R versus NR groups using one-way ANOVA showed no significant difference when comparing total number of patients, number of studies included, heterogeneity via I2, impact factor or AMSTAR. MANOVA analysis of the same group revealed no difference.

    Of papers that gave a recommendation (n=227), 64% were firm (R–F; n=145, 52% of papers overall) and 36% were tempered (R–T; n=82). Of papers that gave no recommendation (n=51), 31% were firm (NR–F; n=16) and 69% were tempered (NR–T; n=35). Within the M group, 58% were R–F (n=114), 29% were R–T (n=55), 9% NR–T (n=18) and 4% NR–F (n=8). Within S, 38% were R–F (n=18), 31% were R–T (n=15), 21% were NR–T (n=10) and 10% NR–F (n=5). For MS, 37% were R–F (n=13), 34% R–T (n=12), 20% NR–T (n=7) and 9% NR–F (n=3). A χ2 test was performed, and a significant relationship was found between the intervention type and the level of conclusiveness of the paper (χ2 (6, n=278)=14.493, P=0.025). Comparison of individual groups using χ2 with a Bonferroni correction (α=0.017) revealed a significant difference between S and M (P=0.014) but not between S and MS (P=0.987) nor M and MS (P=0.065). The number of equivocal reviews (NR–T + NR–F) covered 355 papers and 104 160 patients in M, 503 papers and 385 898 patients in S and 124 papers and 15 371 patients in MS.

    Financial support of the papers audited is detailed in table 3. Notably, government funding was identified as the major sponsor in 22% of M (n=42), 2% of S (n=1) and 11% of MS (n=4). Academia was the primary sponsor in 28% of M (n=55), 44% of S (n=21) and 45% of MS (n=16). The funding source was unclear in 22% of M (n=43), 37% of S (n=17) and 17% of MS (n=6). Comparison of individual groups using χ2 with a Bonferroni correction (α=0.017) revealed a significant difference between S and M on government funding (P<0.001) but not within categories of corporate, academic, combination sponsorship or where the funding was unclear. The MS group was not significantly different from either group across all categories.

    View this table:
    Table 3

    Financial support of papers

    Discussion

    This paper has examined the differences in the level of equivocation between surgical and medical meta-evidence. To our knowledge, this is the first such comparison. We believe it is important to address this issue as meta-evidence continues to be produced in increasing numbers in both medicine and surgery.27 28 While the utility of meta-evidence within medicine is widely acknowledged, surgical interventions are typically more complex and heterogeneous, making the generation of robust surgical meta-evidence difficult.8 9 11 Although the justification for meta-evidence within surgery is weaker than in medicine, the academic cache is transferrable; that is, it maintains its premier position in the busy clinician’s evidence heuristic.

    Papers that could not make a recommendation for practice were more likely to involve a surgical therapy. Papers in the S group were 2.5 times more likely than M papers to be equivocal. MS papers were twice as likely. The only other comparator that was predictive on a paper’s conclusiveness was the number of patients included. On metrics of methodology, number of included studies, heterogeneity and impact factor, there was no difference on univariate or multivariate analysis.

    Surgical meta-evidence was also less likely than medical meta-evidence to be confident in its recommendations for clinical practice, by a factor of two, and more likely to be completely uncertain by a factor of three. In a combined medical and surgical paper, the ratios for these criteria were 1.6 and 2, respectively.

    Previous studies have found that surgical meta-evidence is more likely to have poorer methodology,12 though this paper did not find support for that claim (potentially demonstrating an improving methodology in surgical meta-evidence, a topic for further research). Despite parity on this and other metrics, our study has found that combined surgical evidence is more than twice as likely to be equivocal when compared with corresponding medical reviews. An important distinction to bear in mind here is that AMSTAR assesses the methodology of the meta-analysis or systematic review technique, as opposed to the quality of the original input papers. Audits of original research methodology have found surgical papers to be poorer than medical ones in that regard.29 Reasons for this have been well espoused elsewhere.30 This audit, by focusing on the ability of meta-evidence to provide a recommendation, raises two questions: first, given the prior probability of a clinical recommendation within surgical meta-evidence is 2.5 times less than in medical literature, is aggregate analysis of surgical evidence a worthwhile investment of limited resources?, and second, in light of this, should meta-evidence in surgery still be regarded as the ‘best’ available evidence?

    The purpose of aggregation in level 1 evidence is to maximise our approximation of reality, but considering the findings shown here, is it possible that in surgical meta-literature, where input quality is poorer, aggregation leads to attenuation? High-quality trials will always be well regarded, but one wonders as to the influence of suboptimal trials and equivocal meta-evidence on the acceptance and application of evidence-based surgery. In this setting, a challenge is created for any surgeon attempting to practice ‘best evidence’. This is perhaps best reflected when one looks at the degree of confidence that the authors of each paper have shown in their conclusions; higher levels of uncertainty are expressed in clinical recommendations for surgical procedure when compared with medical therapies by a factor of two.

    Great effort, intellect and perseverance have given us the present surgical evidence and reviews on IBD, but the results of the present study suggest a higher level of scepticism towards surgical evidence and meta-evidence may be warranted. The lack of difference across the metrics studied in this paper, save for type of intervention (surgical vs medical), suggests an unresolved challenge to successfully combining original surgical research. An increase in error appears to be associated with the surgical research process when compared with equivalent medical research, which is exacerbated when combined analysis is performed. Continuation of surgical research that is of inferior quality to medical research, with less predictive power in the meta-evidence setting, weakens the standing of evidence-based surgical practice. However, equally, so too does surgical meta-evidence that must equivocate when presented with the available literature and whose calls for improved methodology in original studies have not been sufficiently heeded, excellent examples of which may be seen in sequential Cochrane reviews.31 32

    Our financial analysis reveals a striking discrepancy in funding between surgical and medical meta-evidence, most notably in the government sector. This is despite a quarter-of-a-billion surgical cases worldwide annually.33 How may these funding shortfalls, compounding the unique challenges of surgical research, be addressed? And in doing so, how may we create a surgical output more cohesive and clinically useful? The role of the international community of surgical academia to address this issue is paramount. In addition to petitioning government, increased levels of collaboration and consolidation may prove valuable.34 Resources may be used in a more focused manner; for instance, the publication requirements of those who aspire to become academic surgeons provides a ready example of a resource that could be used more effectively towards targeted scientific questions.34 Lastly, surgical journals must continue to insist on higher levels of methodology in surgical trials and a greater degree of focus on uniformity of trial design,35 enhancing the reputation of surgical science and hence the argument for funding.

    Strengths and limitations

    The strengths of this ‘overview-of-reviews’ are the large sample of papers, the use of multiple independent reviewers and the validity of AMSTAR as a quality assessment tool. The methods used in search and data-retrieval has been clearly outlined, with explicit inclusion and exclusion criteria.

    The limitations of the study include the inability of AMSTAR to discriminate between poor methodological quality of a study and poor reporting quality within the paper (internal validity). The use of IBD as a framework may introduce selection bias, particularly given that surgical intervention typically represents a failure of medical therapy in IBD. The findings of this ‘review-of-reviews’ are limited in their application outside of IBD research. Similar studies in differing fields will provide a useful basis for comparison. The assessment of a paper’s level of equivocation is subjective and open to bias. An author’s bias towards a subject may also contribute to a paper’s self-reported level of equivocation and the reasons for equivocation. Subtle changes in the language may influence the perceived level of confidence and the rationale for uncertainty.

    Conclusion

    This paper has demonstrated that surgical meta-evidence within the IBD domain is 2.5 times more likely than medical meta-evidence to be unable to provide a recommendation for clinical practice. Whether the intervention being assessed was surgical or medical was the only significant predictor of equivocation when considered against meta-evidence methodology, number of papers, number of patients or level of data heterogeneity. Surgical research also experiences resource limitations where compared with medical research, notably in government funding. We suggest that a discussion should be undertaken within the surgical community, including in this and other journals, about the evolution of the surgical research paradigm; how best to design a system of hypothesis testing that will generate robust results from the unique clinical, moral and human environment of the surgical intervention.

    References

    1. 1.↵
      2. Egger M ,
      3. Ebrahim S ,
      4. Smith GD
      . Where now for meta-analysis? Int J Epidemiol 2002;31:1–5.doi:10.1093/ije/31.1.1
      OpenUrlCrossRefPubMedWeb of Science
    2. 2.↵
      2. Sacks HS ,
      3. Berrier J ,
      4. Reitman D , et al
      . Meta-analyses of randomized controlled trials. N Engl J Med 1987;316:450–5.doi:10.1056/NEJM198702193160806
      OpenUrlCrossRefPubMedWeb of Science
    3. 3.↵
      OCEBM Levels of Evidence Working Group. The Oxford 2011 levels of evidence. Oxford, UK: Oxford Centre for Evidence-Based Medicine, 2011.
    4. 4.↵
      2. Petticrew M
      . Systematic reviews in the social sciences: a practical guide. Boston: Blackwell Publishing, 2006.
    5. 5.↵
      2. Shemilt I ,
      3. Mugford M ,
      4. Vale L , et al
      . Evidence synthesis, economics and public policy. Res Synth Methods 2010;1:126–35.doi:10.1002/jrsm.14
      OpenUrl
    6. 6.↵
      2. Brooke BS ,
      3. Nathan H ,
      4. Pawlik TM
      . Trends in the quality of highly cited surgical research over the past 20 years. Ann Surg 2009;249:162–7.doi:10.1097/SLA.0b013e31819291f9
      OpenUrlCrossRefPubMed
    7. 7.↵
      2. Buchwald H
      . Surgical procedures and devices should be evaluated in the same way as medical therapy. Control Clin Trials 1997;18:478–87.doi:10.1016/S0197-2456(96)00114-6
      OpenUrlCrossRefPubMedWeb of Science
    8. 8.↵
      2. Ergina PL ,
      3. Cook JA ,
      4. Blazeby JM , et al
      . Challenges in evaluating surgical innovation. Lancet 2009;374:1097–104.doi:10.1016/S0140-6736(09)61086-2
      OpenUrlCrossRefPubMedWeb of Science
    9. 9.↵
      2. McLeod RS ,
      3. Wright JG ,
      4. Solomon MJ , et al
      . Randomized controlled trials in surgery: Issues and problems. Surgery 1996;119:483–6.doi:10.1016/S0039-6060(96)80254-6
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      2. Rangel SJ ,
      3. Kelsey J ,
      4. Henry MC , et al
      . Critical analysis of clinical research reporting in pediatric surgery: justifying the need for a new standard. J Pediatr Surg 2003;38:1739–43.doi:10.1016/j.jpedsurg.2003.08.033
      OpenUrlCrossRefPubMed
    11. 11.↵
      2. Hall JC ,
      3. Platell C ,
      4. Hall JL
      . Surgery on trial: an account of clinical trials evaluating operations. Surgery 1998;124:22–7.doi:10.1016/S0039-6060(98)70070-4
      OpenUrlPubMedWeb of Science
    12. 12.↵
      2. Delaney J ,
      3. Laws P ,
      4. Wille-Jørgensen P , et al
      . Inflammatory bowel disease meta-evidence and its challenges: is it time to restructure surgical research? Colorectal Dis 2015;17:600–11.doi:10.1111/codi.12882
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Dellinger EP
      . Increasing inspired oxygen to decrease surgical site infection: time to shift the quality improvement research paradigm. JAMA 2005;294:2091–2.doi:10.1001/jama.294.16.2091
      OpenUrlCrossRefPubMedWeb of Science
    14. 14.↵
      2. Sellke FW ,
      3. DiMaio JM ,
      4. Caplan LR , et al
      . Comparing on-pump and off-pump coronary artery bypass grafting: numerous studies but few conclusions: a scientific statement from the American Heart Association council on cardiovascular surgery and anesthesia in collaboration with the interdisciplinary working group on quality of care and outcomes research. Circulation 2005;111:2858–64.doi:10.1161/CIRCULATIONAHA.105.165030
      OpenUrlAbstract/FREE Full Text
    15. 15.↵
      2. Lee E ,
      3. Dobbins M ,
      4. Decorby K , et al
      . An optimal search filter for retrieving systematic reviews and meta-analyses. BMC Med Res Methodol 2012;12:51.doi:10.1186/1471-2288-12-51
      OpenUrlCrossRefPubMed
    16. 16.↵
      The Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions: Wiley-Blackwell, 2011.
    17. 17.↵
      2. Talley NJ ,
      3. Abreu MT ,
      4. Achkar JP , et al
      . An evidence-based systematic review on medical therapies for inflammatory bowel disease. Am J Gastroenterol 2011;106(Suppl 1):S2–25.doi:10.1038/ajg.2011.58
      OpenUrlCrossRefPubMed
    18. 18.↵
      2. Cima RR ,
      3. Pemberton JH
      . Medical and surgical management of chronic ulcerative colitis. Arch Surg 2005;140:300–10.doi:10.1001/archsurg.140.3.300
      OpenUrlCrossRefPubMedWeb of Science
    19. 19.↵
      2. Fichera A ,
      3. Michelassi F
      . Surgical treatment of Crohn’s disease. J Gastrointest Surg 2007;11:791–803.doi:10.1007/s11605-006-0068-9
      OpenUrlCrossRefPubMedWeb of Science
    20. 20.↵
      2. Jones DW ,
      3. Finlayson SR
      . Trends in surgery for Crohn’s disease in the era of infliximab. Ann Surg 2010;252:307–12.doi:10.1097/SLA.0b013e3181e61df5
      OpenUrlCrossRefPubMed
    21. 21.↵
      2. Quan H ,
      3. Parsons GA ,
      4. Ghali WA
      . Validity of procedure codes in International Classification of Diseases, 9th revision, clinical modification administrative data. Med Care 2004;42:801–9.doi:10.1097/01.mlr.0000132391.59713.0d
      OpenUrlCrossRefPubMedWeb of Science
    22. 22.↵
      2. Shea BJ ,
      3. Grimshaw JM ,
      4. Wells GA , et al
      . Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.doi:10.1186/1471-2288-7-10
      OpenUrlCrossRefPubMed
    23. 23.↵
      2. Shea BJ ,
      3. Hamel C ,
      4. Wells GA , et al
      . AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009;62:1013–20.doi:10.1016/j.jclinepi.2008.10.009
      OpenUrlCrossRefPubMedWeb of Science
    24. 24.↵
      Thompson Reuters. Journal citation reports. JCR science, 2012.
    25. 25.↵
      Microsoft Corporation. Microsoft excel for Mac. 14.3.2 edn: Microsoft Corporation, 2011.
    26. 26.↵
      IBM Corp. IBM SPSS statistics for macintosh. 24.0 edn: IBM Corp, 2016.
    27. 27.↵
      2. Bastian H ,
      3. Glasziou P ,
      4. Chalmers I
      . Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010;7:e1000326.doi:10.1371/journal.pmed.1000326
    28. 28.↵
      2. Tebala GD
      . What is the future of biomedical research? Med Hypotheses 2015;85:488–90.doi:10.1016/j.mehy.2015.07.003
      OpenUrl
    29. 29.↵
      2. Sinha S ,
      3. Sinha S ,
      4. Ashby E , et al
      . Quality of reporting in randomized trials published in high-quality surgical journals. J Am Coll Surg 2009;209:565–71.doi:10.1016/j.jamcollsurg.2009.07.019
      OpenUrlCrossRefPubMedWeb of Science
    30. 30.↵
      2. Rosenthal R ,
      3. Kasenda B ,
      4. Dell-Kuster S , et al
      . Completion and publication rates of randomized controlled trials in surgery: an empirical study. Ann Surg 2015;262:68–73.doi:10.1097/SLA.0000000000000810
      OpenUrlCrossRefPubMed
    31. 31.↵
      2. Lustosa SA ,
      3. Matos D ,
      4. Atallah AN , et al
      . Stapled versus handsewn methods for colorectal anastomosis surgery. Cochrane Database Syst Rev 2001:CD003144.doi:10.1002/14651858.CD003144
    32. 32.↵
      2. Neutzling CB ,
      3. Lustosa SA ,
      4. Proenca IM , et al
      . Stapled versus handsewn methods for colorectal anastomosis surgery. Cochrane Database Syst Rev 2012:CD003144.doi:10.1002/14651858.CD003144.pub2
    33. 33.↵
      2. Weiser TG ,
      3. Regenbogen SE ,
      4. Thompson KD , et al
      . An estimation of the global volume of surgery: a modelling strategy based on available data. Lancet 2008;372:139–44.doi:10.1016/S0140-6736(08)60878-8
      OpenUrlCrossRefPubMedWeb of Science
    34. 34.↵
      2. Søreide K ,
      3. Alderson D ,
      4. Bergenfelz A , et al
      . Strategies to improve clinical research in surgery through international collaboration. Lancet 2013;382:1140–51.doi:10.1016/S0140-6736(13)61455-5
      OpenUrlCrossRefPubMedWeb of Science
    35. 35.↵
      2. Wynne KE ,
      3. Simpson BJ ,
      4. Berman L , et al
      . Results of a longitudinal study of rigorous manuscript submission guidelines designed to improve the quality of clinical research reporting in a peer-reviewed surgical journal. J Pediatr Surg 2011;46:131–7.doi:10.1016/j.jpedsurg.2010.09.077
      OpenUrlPubMed

    Footnotes

    • Contributors JDD and AFE were the designers of the work. The acquisition of the data was performed by JDD, JTH, RKD and PJL. JDD and AFE contributed to the analysis and interpretation of the data. The work was drafted by JDD, with critical revision by JTH, RKD, PJL and AFE. All authors gave final approval for the published version and agreed to be held to its accuracy and integrity.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement There are no unpublished data for this study. Any enquiries relating to the paper are welcome via email: jdel2642@uni.sydney.edu.au.