Article Text
PDF
Abstract
Objective To provide a comprehensive review of registered COVID-19-related randomised controlled trials (RCTs) in mainland China and evaluate the transparency of reporting through comparison of registrations, protocols and full reports.
Design Systematic review of trial registrations and publications.
Data sources International Clinical Trials Registry Platform, Chinese Clinical Trial Registry, ClinicalTrials.gov, the ISRCTN registry and EU Clinical Trial Register were accessed on 1 February 2022. Publications were searched in PubMed, Embase, Cochrane Library, Google Scholar, CNKI.net and Wanfangdata from 10 February 2022 to 12 February 2022.
Eligibility criteria Eligible trials were COVID-19 related RCTs carried out in mainland China. Observational studies, non-randomised trials and single-arm trials were excluded.
Data extraction and synthesis Two reviewers independently extracted data from registrations, publications and performed risk of bias assessment for trial reports. Information provided by registrations and publications was compared. The findings were summarised with descriptive statistics.
Results The number of eligible studies was 415. From these studies 20 protocols and 77 RCT reports were published. Seven trials published both protocol and RCT full report. Between registrations and publications, discrepancy or omission was found in sample size (7, 35.0% for protocols and 47, 61.0% for reports, same below), trial setting (13, 65.0% and 43, 55.8%), inclusion criteria (12, 60.0% and 57, 74.0%), exclusion criteria (10, 50.0% and 54, 70.1%), masking method (9, 45.0% and 35, 45.5%) and primary outcome or time frame of primary outcome measurement (14, 70.0% and 51, 66.2%). Between protocols and full reports, 5 (71.4%) reports had discrepancy in primary outcome or time frame of primary outcome measurement.
Conclusions Discrepancy among registrations, protocols and reports revealed compromised transparency in reporting of COVID-19-related RCTs in mainland China. The importance of trial registration should be further emphasised to enhance transparent RCT reporting.
- COVID-19
- Clinical trials
- STATISTICS & RESEARCH METHODS
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The study provided a full coverage of publicly registered randomised controlled trials (RCTs) related to the prevention, treatment or prognosis of COVID-19 infection.
The study identified publications citing these registration records and examined the consistency of methodology among registrations, protocols and full reports.
The study included only RCTs performed in mainland China.
RCT reports that did not cite any registration number (if any) were not included in the analysis.
Introduction
Evidence-based medicine aims to achieve optimal decision making in the care of individual patients, using the current best evidence.1 Among all types of evidence, randomised controlled trials (RCT), together with systematic reviews of RCTs, are accorded the highest level of credibility and thus play an important role in evidence-based medicine.2 However, not all RCTs are of the same quality, which leads to clinical research methodologists’ emphasis on transparency in RCT reporting.3
‘The whole of medicine depends on the transparent reporting of clinical trials.’4 Transparent reporting requires a complete description of methodology through which the trial data are collected and analysed, a report without omitting any data generated by the trial and a standard way of writing.3 Much effort has been spent on promoting transparent reporting, including the International Committee of Medical Journal Editors (ICMJE) member’s requirement for registration in public trial registry prior to enrollment5 and the announcement of CONsolidated Standards Of Reporting Trials (CONSORT) statement.6 However, the overall transparency of RCTs remains suboptimal, and incomplete or selective reporting in publication remains an issue. Before mandatory clinical trial registration was enforced by ICMJE in 2005, Chan et al reported 62% published trial report had modified, introduced or omitted in comparing the trial protocols approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg7; when similar approach was applied to trials approved for funding by the Canadian Institutes of Health Research, researchers found that primary outcome differed between reports and protocols in 40% of the trials.8
To ensure transparency in trial reporting and reduce selective reporting, the ICMJE initiative encouraged researchers to make trial information available to the public.5 Since then, the number of registrations has increased greatly.9 10 Prospective registration is a powerful tool in reducing selective reporting as it reflects researchers’ intention at planning stage of the trials, and can be compared with published full reports.11 Mathieu et al compared registrations and publications of RCTs in journals with highest impact factors in cardiology, rheumatology and gastroenterology, finding that 31% or properly registered RCTs had discrepancies between registered and published primary outcomes12; according to Rayhill et al, only about 25% of RCTs published in the core headache medicine journals displayed proper compliance with trial registration.13 In two systematic reviews summarising studies that compared registrations with full reports, Jones et al found the median proportion of trials with identified discrepancy in primary outcome was 31%,14 and Li et al also reported high level of inconsistency in outcome reporting ranging from 14% to 100%.11
The ongoing pandemic of COVID-19 is a major public health concern. Numerous clinical trials have been registered and published to address scientific questions regarding the prevention, treatment and prognosis of the disease, and so far, many reports have been published. Kataoka et al examined COVID-19 RCT articles in medRxiv and PubMed, revealing problems in research methods and the impact on report quality associated with accelerated publication.15 However, no study has examined the transparency and selective reporting in COVID-19 trials by comparing full reports with trial registrations. In this study, we aimed to provide a comprehensive review of characteristics of registered COVID-19-related RCTs in mainland China, and evaluate the level of transparency and selective reporting by comparing trial registrations, published protocols and full reports.
Methods
Clinical trial registration screening
International Clinical Trials Registry Platform (ICTRP), Chinese Clinical Trial Registry (ChiCTR), ClinicalTrials.gov (NCT), the ISRCTN registry (ISRCTN) and EU Clinical Trial Register (EUCTR) were accessed on 1 February 2022. A complete list of COVID-19 trials updated on 22 January 2022 was retrieved from ICTRP. Eligible studies were RCTs related to prevention, treatment or prognosis of COVID-19 infection. Studies were excluded if they were observational, non-randomised, single-arm trials or outside of mainland China. For multicentre trials, all centres must be located in mainland China to be eligible for analysis. From ChiCTR, index of studies of COVID-19 updated on 22 December -2021 was obtained. Studies registered after this date were screened manually. From NCT, list of interventional clinical studies related to COVID-19 was accessed, and map panel was used to select studies performed in mainland China. For ISRCTN and EUCTR, searches were conducted manually with keywords “COVID-19”, “COVID-19”, “SARS-Cov-2” or “2019-nCov” and the results were screened for eligibility according to above mentioned criteria (for review protocol and detailed search strategy, see online supplemental file 1).
Supplemental material
We extracted registration ID, date of registration, date of last update, date of first enrolment, scientific/official title, objective, intervention of interest, comparator, primary purpose, recruitment status, estimated enrolment, arms, ethical approval information (ChiCTR only), name and location of centres, randomisation (ChiCTR only), masking, inclusion and exclusion criteria, primary outcomes and other relevant information from registration records. Repeated registration was identified through examining similarity of trial characteristics. For multiregistered trials, only the record cited by publication and (if all/none of repeated registration records was cited) the most recently updated record shall be eligible for further analysis. Characteristics of included registrations were summarised and presented as count (%) or median (IQR). Scatter plot and line chart were plotted to illustrate the trend of registered trials in each province of mainland China.
Literature search
Literatures citing these registrations were searched from 10 February 2022 to 12 February 2022 in PubMed, Embase, Cochrane Library, Google Scholar and two frequently used databases for Chinese language literature, CNKI.net and Wanfangdata, using registration ID as unique identifiers. From published protocols and trial reports, date of first enrolment, estimated enrolment, arms, centre names, inclusion/exclusion criteria, primary/secondary outcomes and information regarding randomisation and masking were extracted. Risk of bias assessment for published trial reports was performed using RoB 2.16 Next, information extracted from registration records, published protocols and reports was compared, and the level of consistency was evaluated.
Registration screening, literature search, data extraction, risk of bias assessment and comparison were independently performed by two reviewers and consensus was reached through discussion.
Patient and public involvement
Patients or public was not involved in the design, conduct, reporting or dissemination plans, since the research question of this study made such involvement unnecessary.
Results
A total of 435 registration records met eligibility criteria, in which 20 studies were repeatedly registered on ChiCTR and NCT, making the number of eligible studies 415 (see online supplemental file 2). The flow chart of screening process is shown in figure 1.
Supplemental material
The screening process of registration records.
Characteristics of registered trials
Among all included studies, 303 (73.0%) were registered in ChiCTR, 111 (26.7%) in NCT and 1 (0.2%) in ISRCTN and 243 (58.6%) were registered before the date of first enrolment. Most of the trials aimed to investigate the prevention (105, 25.3%) or treatment (290, 69.9%) of COVID-19. At the time of the most recent update, 133 (32.0%) trials did not start recruitment, 206 (49.6%) were recruiting and 55 (13.3%) trials completed recruitment, while 21 (5.1%) studies were suspended, terminated or withdrawn. The median estimated enrolment was 120. 32.8% trials (126) were multicentred and 68.4% (284) were two-armed. For participants, 285 (68.7%) trials recruited clinical diagnosed COIVD-19-infected patients, 8 (1.9%) recruited suspected cases or close contacts of diagnosed patients, and 122 (29.4%) recruited participants not infected by COVID-19. 39.5% (164) registrations did not specify any masking method, 125 (30.1%) were open-label studies and single/double masking was applied in 126 (30.4%) studies. A total of 415 trials registered a total of 2951 outcomes, including 1048 primary outcomes and 1903 secondary outcomes. A total of 186 (44.8%) trials specified 1 primary outcome in registration record and 226 (54.5%) declared more than one primary outcomes. Primary outcome was missing for 3 (0.9%) registrations. 313 (75.4%) registrations had prespecified secondary outcomes (table 1).
Characteristics of all included studies
Charts were plotted to illustrate the trend of increasing trial registration in mainland China and each Province. The first COVID-19 trial in mainland China was registered on 23 January 2020. As the pandemic began in Wuhan, COVID-19-related RCTs emerged in Hubei province first, and then expanded to other provinces. The next 4 months witnessed dramatic increase of trial registrations. this trend reached plateau around June 2020 and was thereafter steadily increasing until the day of data extraction (figure 2A).
(A) The trend of registration of COVID-19-related RCTs in total and in each Province. When a trial is multicentred and recruited participants in multiple provinces, the trial is considered to take place in these provinces simultaneously. The publication status of these registered trials is also shown. (B) Results of risk of bias assessment are shown for each RCT report included in this review, the risks in each domain and overall risk are also summarised. RCT, randomised controlled trial.
Publications from registered trials
From the 415 registered trials, 85 reports and 20 protocols were published; eight reports were excluded from analysis since the study design was described as non-RCT (see online supplemental file 1). Risk of bias assessment was performed for 77 RCT reports. Overall, high risk was found in 35 (45.5%) reports; 33 (42.9%) reports had some concerns; 9 (11.7%) were at low risk. The most common risk of bias was selection of reported result, with 60 (77.9%) reports having some concerns or high risk. Deviations from intended interventions (43, 55.9% for studies with some concerns or high risk, same below) and randomisation issues (31, 40.3%) were also frequently documented. (figure 2B)
Comparison between protocols and trial registrations
In 20 published protocols, 6 (30.0%) had deviation in sample size and 8 (40.0%) had discrepancy in setting; More than half of them differed in inclusion criteria (12, 60.0%) or exclusion criteria (10, 50.0%); 9 (45.0%) did not specify who was masked in the text or registration. There were 11 (55%) discrepancies in nature of the primary outcomes between the registrations and the protocols. This included introducing new primary outcomes in protocol (8, 40.0%), omitting registered primary outcomes (8, 40.0%) and describing registered primary outcomes as secondary outcomes (9, 45.5%). None of the protocols declared outcome change and rationale. Only two (10.0%) protocols were consistent in all domains (table 2) (see online supplemental file 2).
Comparison between registration records and publications
Comparison between reports and trial registrations
From 74 registrations, 77 RCT reports eligible for analysis were published. Forty-seven (61.0%) of the reports had discrepancies in sample size and the deviation was more than 20% in 36 (46.8%) of them. Discrepancy in setting was present in 21 (27.3%) reports. Fifty-four (70.1%) reports had deviation in inclusion criteria and 46 (59.7%) had different exclusion criteria. Seven (9.1%) reports had deviation in masking method, among which masking was upgraded (open-label changed to single/double-blind or single-blind changed to double-blind) in five (6.5%) studies and downgraded in two (2.6%) study. In primary outcome, 4 (5.2%) reports did not specify primary outcome in text or corresponding registration; 34 (44.2%) reports had deviation in primary outcome, including introducing new primary outcomes (18, 23.4%), omitting registered primary outcomes (23, 29.9%), describing registered secondary outcomes as primary outcomes (7, 9.1%) and describing registered primary outcomes as secondary outcomes (15, 19.5%). Furthermore, seven (9.1%) reports did not specify time frame of primary outcome measurement in registry/text and six (7.8%) had deviation in time frame of primary outcome measurement. None of the reports declared outcome change and rationale. Only four (5.2%) reports maintained full consistency in all domains (table 2). Among the 183 primary outcomes of the 74 registrations, 59 (32.2%) were correctly reported (see online supplemental file 2).
Comparison between protocols and full reports
There were seven trials where both protocol and RCT report were published. When comparing full reports to protocols, deviation in sample size and setting was found in six (85.7%) and three (42.9%) reports, respectively. More than half of the reports four (57.1%) differed from protocols in inclusion criteria and three (42.9%) differed in exclusion criteria. Deviation in masking was found in one (14.3%) report. Five (71.4%) reports had difference in primary outcome or time frame of primary outcome. None of the reports disclosed outcome change and rationale. None of the reports maintained full consistency with the protocols (table 3) (see online supplemental file 2).
Comparison between protocols and full reports
Discussion
To summarise, 20 protocols and 77 RCT reports were published from 415 registrations in mainland China. Comparing to registrations, 90% protocols and 94.8% reports had discrepancy in at least one domain, in which deviation in primary outcome or time frame of primary outcome measurement occurred in 60% (12/20) protocols and 51.9% (40/77) reports. In trials where both protocol and report were published, 71.4% (5/7) full reports had discrepancies in primary outcome or time frame of measurement comparing to the protocols. Furthermore, risk of bias assessment revealed that a majority (88.4%) of the RCT reports had some concerns or high risk in overall bias, and selection of reported result was the most prevalent issue.
Since the pandemic of COVID-19 is a major public health issue, RCT evidence is urgently needed to support clinical decision making in the prevention, treatment or prognosis of this disease. However, methodological concerns in such trials were noted. Previously, Kataoka et al reported research methods and reporting problems in medRxiv and PubMed publications related to COVID-19 RCTs, and inconsistency with trial registration was identified in 62% of 13 medRxiv literatures and 30% of 16 PubMed articles; the authors pointed out that these problems might result from the accelerated publication process,15 yet the study only covered a limited number of publications because it was performed in June 2020. One year and a half later, our comprehensive review of COVID-19 trial registrations, protocols and reports suggested the finding was not incidental.
To assess the significance of our findings, we compared the results with other studies that evaluated the consistency among trial registrations, protocols and reports. In 2011, Dwan et al systematically reviewed cohort studies comparing contents of trial registry entries or protocols with trial reports, and in 33% to 67% of RCTs primary outcome was the same in protocol as in publication, while in 69%–82% of RCTs primary outcome in report was consistent with trial registration17; in the study of Li et al, the median inconsistency of outcome reporting was 54% (IQR: 29%–72%)11; Jones et al reported a median proportion of 31% trials had discrepancy between registered and published primary outcome. The rates of discrepancy reported by our study were higher than Dwan et al or Jones et al studies, and were similar to the median rate reported by Li et al.
These results revealed compromised transparency in COVID-19 related RCT reporting in mainland China, and also suggested the presence of selective reporting in RCTs in other fields of study. Such findings are noteworthy especially for peer-reviewers and journal editors, as compromised transparency might undermine the value of these trials. Furthermore, the fact that RCT reports were published by less than a quarter of registered trials indicated potential presence of publication bias: a phenomenon of selective publication of studies depending on the results.18 However, since our research methods were not designed to detect publication bias, we decided to leave this question to future research.
ICMJE’s policy of mandatory prospective registration in member journals led to dramatic increase of trial registrations, while the registration data were often inadequate, changed over time and differed between registration and publication.19 The goal of ICMJE’s policy was to promote transparent reporting of trials, yet this could not be achieved unless journal editors and reviewers fully utilise information provided by trial registration: a survey by Mathieu et al suggested only around one-third reviewers routinely used registration information when evaluating manuscripts.20 Changes in primary outcome or other domains of trial design might be due to either good reasons or investigators’ effort to produce favourable results from the data.14 In either case, the validity of evidence provided by the trial could not be confidently assessed without emphasising transparency of reporting. Cooperation of different stakeholders is required to promote transparency. We suggest principal investigators should ensure that the trial is prospectively registered and registration information is properly filled; peer reviewers should routinely use trial registries to assess manuscripts and demand explanation whenever discrepancy occurs; journal editors should prioritise the evaluation of trial registration in peer review process.11 14 Future studies will be needed to reveal the trend of consistency over time and assess possible improvement caused by increasing scrutiny of peer reviewers, new policy of journals and ascensive familiarity of investigators with trial registration process.
The study had several limitations. First, the complete lists of COVID-19-related clinical trials were retrieved from ICTRP and ChiCTR and filtered by the labels provided by these lists, thus, the completeness and precision of record screening relied on the correctness of these lists. Second, the search of publications for eligible registration record was conducted with trial registration number as unique identifiers; if a publication did not cite any registration number, the article could not be included in analysis. This approach might lead to omission of publications that did not contain any trial identifier, but provided definitive evidence in linking registrations to publications and minimised the possibility of making mistakes. Third, the reviewers only analysed the most recent updated version of registration record, without considering the historical changes, yet the validity of conclusion of this review is not jeopardised, since this approach tended to overestimate, but not underestimate, the overall transparency, in that registrations are often modified after trial completion to display false consistency with publications.14 19 Last, our search results were limited to peer-reviewed English and Chinese literatures, and the findings could not reliably represent studies published in languages other than English or Chinese. We did not review any literatures that had not undergone peer-review process (eg, manuscripts posted on preprint servers). However, without quality control in review process, it would be reasonable to assume that such preprints have no lower rate of discrepancy compared with peer-reviewed publications.
Conclusion
The high rates of discrepancy among registrations, protocols and full reports of COVID-19-related RCTs in mainland China revealed compromised transparency in trial reporting. Investigators, peer reviewers and journal editors should make efforts to improve the utilisation of trial registration information and promote transparent reporting.
Data availability statement
All data relevant to the study are included in the article or uploaded as online supplemental information.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
The authors would like to thank the reviewers for their invaluable suggestions.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Both reviewers (YC and RY) contributed equally in conceptualisation, data extraction, analysis, visualisation and drafting the manuscript. Both reviewers read and approved this manuscript. RY acted as the guarantor of the study.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.