Article Text
PDF
Abstract
Introduction Reporting guidelines are important tools for improving the quality of medical research. The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s Library contains a comprehensive and up-to-date database of reporting guidelines relevant to health research. Only 31% of reporting guidelines published from 2010 to 2014 reported using the Delphi technique, and the reporting quality of the Delphi technique in reporting guidelines is unknown even though the use of the Delphi technique was recommended in the guidance for reporting guidelines. We will assess the quality reports of the Delphi technique or modified Delphi technique in reporting guidelines.
Methods and analysis The present study is a systematic analysis of the EQUATOR Network Library. We will include all reporting guidelines in the EQUATOR Network that used the Delphi technique or modified Delphi technique, published since 1 January 2011 and registered in the EQUATOR on or before 31 May 2018. Our primary outcome is the reporting quality of the Delphi technique, measured by the quality score (each item) in the Delphi technique. We will also examine the relationship between the reporting quality score (each item) of the Delphi technique and year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published in open access policy.
Ethics and dissemination Ethics approval will not be applicable for this study. This protocol has been registered in the University Hospital Medical Information Network Clinical Trials Registry. We will publish our findings in a peer-reviewed journal and may also present them at conferences.
Trial registration number UMIN000032685.
- Delphi technique
- reporting guidelines
- systematic analysis
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first study to investigate the factors associated with each item in the quality score of the Delphi technique in reporting guidelines.
We will perform an independent assessment for reporting guidelines.
Applicability will be limited because the analyses investigating the quality score of the Delphi technique will include only reporting guidelines registered in the Enhancing the QUAlity and Transparency Of health Research Network Library and will not include other, possibly low-quality, reporting guidelines.
We will not investigate whether the reporting quality of the Delphi technique in reporting guidelines affects the reporting quality of individual studies referring to these reporting guidelines.
Introduction
Critical appraisal and effective dissemination of research is hindered by insufficient reporting of a study’s methodology and findings.1 Additionally, insufficient reporting impedes the applicability and misrepresents results used by patients and practitioners.1 To improve the quality of research, experts have developed reporting guidelines.2 Reporting guidelines are important tools for improving the quality of medical research.2 The number of reporting guidelines in the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s Library has increased. The Network’s site contains a comprehensive database of reporting guidelines for health research.3 There are almost 400 reporting guidelines in the EQUATOR Network.4
The three main formal consensus methods used in the health field are the Delphi technique, Nominal Group Technique (NGT) and consensus development conferences.5 The Delphi technique is widely applied in order to obtain input from a group of experts.6–8 The method is characterised by anonymity between members with structured feedback.6 9 Participants may regulate their initial ratings based on feedback from the group in a number of accompanying loops.6 9 The Delphi technique consists of any type of self-administered questionnaire with no meetings, whereas the modified Delphi technique consists of the use of a self-administered questionnaire, combined with a physical meeting of the experts, to discuss the results or rate the indicators.10 11 The Delphi method and the NGT are associated with obtaining a group decision from a suite of experts.5 On the other hand, consensus development conferences have the further aim of preparing a public forum for discussion.5
The NGT and consensus development conferences have limitations. The NGT has a smaller number of participants than does the Delphi technique, with the potential for dominant participants to inordinately affect the group.12 Consensus development conferences depend on implicit methods (qualitative or simple quantitative methods such as majority voting), while the Delphi method and the NGT practice explicit, statistical integration in order to combine the judgements of experts.5
A previous study suggests the reporting of consensus methods in reporting guidelines should be improved.2 Exercising the Delphi technique in reporting guideline development is important because of its potential to add participants in the reporting guideline development process3 and reduce variance of opinion within the group between the two rounds,13 in addition to having higher between-group reliability ratings than NGT.14 Therefore, the technique will improve the quality of guidelines.3 13 14 Only 13% of reporting guidelines published from 2010 to 2014 included the use of the Delphi technique2 even though the use of the Delphi consensus method was recommended in the guidance for reporting guidelines.3 The study, however, did not assess the reporting quality of the Delphi technique among the reporting guidelines, and this aspect remains unknown.2
Objectives
We will assess the quality of reports of the Delphi technique or the modified Delphi technique in reporting guidelines.
Methods and analysis
Types of studies to be included
We will include reporting guidelines in the EQUATOR Network that used the Delphi technique or modified Delphi technique, were published since 1 January 2011, and registered in the EQUATOR Network on or before 31 May 2018. We will select reporting guidelines published on and after January 2011 because a previous study that recommends using the Delphi technique in reporting guideline development was published in February 2010.3 We will only include the most recent versions of reporting guidelines in order to eliminate duplication in the guidelines. We will check for the recent versions of guidelines by screening for the data item ‘previous versions of this guideline/guideline history’ in each of the reporting guidelines in the EQUATOR Network. We will also check whether meetings were held between Delphi rounds. We will regard the reporting guidelines as ‘Delphi’ when meetings were not held between Delphi rounds and as ‘modified Delphi’ when meetings were held between Delphi rounds.
Search methods
We will search the EQUATOR Network Library after 31 May 2018. The search will be subjected to English language restrictions.
Study selection
One of three authors (MB, YT and YK) will assess the eligibility based on a full-text review of reporting guidelines identified by the initial search and another author (MB, YT or YK) will confirm the contents. We will search using the terms ‘Delphi’ or ‘modified Delphi’ in the text and check whether the Delphi technique or modified Delphi technique was used. We will resolve disagreements by discussion between the authors (MB, YT and YK).
Data extraction and assessment
For each of the included reported guidelines, one author (MB) will extract the Delphi technique information and another author (YT or YK) will confirm the contents. We will resolve disagreements by discussion between the authors (MB, YT and YK).
We will extract data for the four quality score items in the Delphi technique.15 A recent study proposed a quality score in the Delphi technique after assessing quality in reports of the Delphi technique, published in 2000–2009.15 The four items are as follows:
Were criteria for participants reproducible? (Yes or No): The method to select and exclude participants is stated. Number and type of participant subgroups (eg, patients, generalists and experts) are needed.
Was the number of rounds to be performed stated? (Yes or No): We will categorise this as ‘Yes’ when the number of rounds is stated in the methods or results. We will categorise this as ‘Yes’ when researchers report the actual number of Delphi rounds in the results.
Were criteria for dropping items clear? (Yes, No or Not applicable): The prespecified criteria for dropping items at each round are reported.
Are stopping criteria, other than rounds, specified? (Yes or No): The prespecified criteria for stopping the Delphi process, other than a statement of the number of rounds, are reported. For example, the prespecified criteria are related to the consensus or stability of responses.
We will score ‘Yes’ and ‘Not applicable’ as 1 and score ‘No’ as 0, as done in the previous study.15 Two authors (MB and YT or YK) will independently assess the score for each reporting guideline. We will adopt the median of the quality score in the earliest full publication in case of multiple publications with the same guideline.
We will extract the following factors potentially associated with the reporting quality of the Delphi technique: year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy. One author (MB) will extract data for the impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy and another author (YT or YK) will confirm the contents. We will resolve disagreements by discussion between the authors (MB, YT and YK). YK will perform web scraping from PubMed and Google Scholar with Python V.3.6 (Python Software Foundation) and collect data for the year of publication, and number of authors. We will record the number of editors as the number of authors if some of the reporting guidelines are books or handbooks. We will define funding as the receipt of any supporting funds for conducting the research. We will regard sources of funding as ‘industry’ when funds are received from an industry (for example, pharmaceutical companies).16 We will regard sources of funding as ‘non-industry’ when the funds are from government, other academic or non-profit organisations.16 We will define multiple publications as publications in multiple journals.3 We will regard translated versions of original guidelines as multiple publications if the translated versions are published in journals. We will check for multiple publications by screening for the data item ‘full bibliographic reference’ of each of the reporting guidelines in the EQUATOR Network. The year of publication will be the oldest year when there are multiple publications for reporting guidelines. We will extract the impact factors determined by the 2018 Journal Citation Reports. An impact factor will be a mean value of multiple publications when there are multiple publications for reporting guidelines. We will deem that the guidelines are published according to open access policy when at least one full-text of the guidelines is available on the web (whether the full-text is downloadable will not be considered). We will check the official sites of 15 reporting guidelines, which are highlighted as ‘Reporting guidelines for main study types’ in the EQUATOR Network and collect additional information about year of publication, impact factor and multiple publications, as well as whether the guidelines are published according to open access policy. We will contact the corresponding authors of the reporting guidelines for additional information if necessary.
Our primary outcome of interest will be the reporting quality of the Delphi technique (each item) in the reporting guidelines. We will also examine the relationship between the reporting quality score (each item) of the Delphi technique and year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy.
Patient and public involvement
We evolved the study protocol without patient participation. This study will use only public data without patient recruitment. We will spread the results via web sites and social network services to patients and the public.
Sample size
The sample size calculation for a primary outcome will not be applicable because the sample size of the reporting guidelines is determined beforehand.
Data analysis
We will report the frequency of the reporting quality score (each item) of the Delphi technique as the descriptive analysis. We will explore correlations, using Fisher’s exact test, between each item of the quality score (Yes, No or Not applicable) and the following possible predictors, defined a priori: year of publication, number of authors, impact factor, sources of funding (industry, non-industry), multiple publications and whether the guidelines are published according to open access policy. We will conduct prespecified sensitivity analyses by repeating the analysis and excluding additional data from the official websites of the 15 reporting guidelines.
All p values will be two-sided. P values will be considered statistically significant if less than 0.05. We will not perform an adjustment of the alpha level for multiple comparisons because our study is an exploratory study. Alpha level adjustment is not essential in exploratory analyses.17 All statistical analyses will be performed with EZR (Saitama Medical Center, Jichi Medical University, Saitama, Japan), which is a graphical user interface for R (The R Foundation for Statistical Computing, Vienna, Austria).18 More precisely, it is a modified version of R commander designed to add statistical functions frequently used in biostatistics.
Ethics and dissemination
The planned completion date of the present study is 31 December 2019. We will publish our findings in a peer-reviewed journal and may also present them at conferences.
Discussion
This is the first study to investigate the factors associated with each item in the quality score of the Delphi technique in reporting guidelines.
This study will reveal the quality of reports of the Delphi technique in reporting guidelines. Problems with the quality of reports of the Delphi technique will be detected. Therefore, this study will be potentially used to improve the quality of reports of the Delphi technique in reporting guidelines. Improved reporting guidelines will result in better health research.19 Therefore, this study has the potential to alter the quality of reporting guidelines and provide useful resources in developing reporting guidelines. This study may also result in new recommendations about the quality of reports of the Delphi technique in the development of reporting guidelines.
There are several expected limitations for this study. First, the applicability will be limited because the analyses investigating the quality score of the Delphi technique include only reporting guidelines registered in the EQUATOR Network Library. The Library contains a comprehensive database of reporting guidelines relevant to health research.3 However, other, possibly low-quality, reporting guidelines may be missing. Second, we will not investigate whether the reporting quality of the Delphi technique in reporting guidelines affects the reporting quality of individual studies referring to these reporting guidelines. However, reporting guidelines created in good order may affect the reporting quality of individual studies because a type of reporting guideline, the Preferred Reporting Items for Systematic Reviews and Meta-Analysis, has increased the quality of reporting in individual systematic reviews and meta-analyses.20 Third, there is no information on the reliability and validity of the quality score. However, we will use the score because we contend it represents a necessary first step for assessing the reporting quality of the Delphi technique in the absence of other measures. Fourth, this study is based on an exploratory analysis and will provide information rather than recommendations.
In conclusion, this study will provide a comprehensive investigation about the reporting quality of the Delphi technique in reporting guidelines using the EQUATOR Network. The expected findings will show the needs and key factors for improving the reporting quality of the Delphi technique in reporting guidelines.
Acknowledgments
We would like to thank the EQUATOR Network and the UK EQUATOR Centre for managing the EQUATOR Network. We would also like to thank Dr Hiraku Tsujimoto (Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Hyogo) for the conception and design of the research. We would like to thank Editage (http://www.editage.jp) for English language editing.
References
Footnotes
Contributors MB, YT and YK contributed to the conception and design of the research. MB is fully responsible for writing the protocol. All authors gave final approval of the protocol before submission. After the publication of the protocol, we plan for the following contributions by each author: MB, YT and YK will screen the relevant records of the EQUATOR Network Library and extract data. MB, YT and YK will conduct the data analysis without being blind to the data. MB, YT and YK will write the manuscript.
Funding This work was supported by the Department of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center, Nagoya University Academy of Psychiatry and by self-funding.
Disclaimer The funder plays no role in developing the protocol.
Competing interests MB has received speaker honoraria from Dainippon Sumitomo; honoraria for a manuscript from Seiwa Shoten Co., Ltd. and travel fees from Yoshitomi Pharmaceutical Industries Ltd.
Provenance and peer review Not commissioned; externally peer reviewed.
Patient consent for publication Not required.