Article Text

Original research
Clinical study reports published by the European Medicines Agency 2016–2018: a cross-sectional analysis
  1. David Byrne1,2,
  2. Ciaran Prendergast3,
  3. Tom Fahey1,2,
  4. Frank Moriarty3
  1. 1 Department of General Practice, Royal College of Surgeons in Ireland, Dublin, Ireland
  2. 2 Royal College of Surgeons in Ireland, HRB Centre for Primary Care Research, Dublin, Ireland
  3. 3 School of Pharmacy and Biomolecular Sciences, Royal College of Surgeons in Ireland, Dublin, Ireland
  1. Correspondence to Dr David Byrne; davidbyrne{at}rcsi.ie

Abstract

Objectives To describe the characteristics of clinical study report (CSR) documents published by the European Medicines Agency (EMA), and for included pivotal trials, to quantify the timeliness of access to trial results from CSRs compared with conventional published sources.

Design Cross-sectional analysis of CSR documents published by the EMA from 2016 to 2018.

Methods CSR files and medication summary information were downloaded from the EMA. Individual trials in each submission were identified using document filenames. Number and length of documents and trials were determined. For pivotal trials, trial phase, dates of EMA document publication and matched journal and registry publications were obtained.

Results The EMA published documents on 142 medications that were submitted for regulatory drug approval. Submissions were for initial marketing authorisations in 64.1%. There was a median of 15 (IQR 5–46) documents, 5 (IQR 2–14) trials and 9629 (IQR 2711–26,673) pages per submission, and a median of 1 (IQR 1–4) document and 336 (IQR 21–1192) pages per trial. Of all identified pivotal trials, 60.9% were phase 3 and 18.5% were phase 1. Of 119 unique submissions to the EMA, 46.2% were supported by a single pivotal trial, with 13.4% based on a single pivotal phase 1 trial. No trial registry results were identified for 26.1% trials, no journal publications for 16.7% and 13.5% of trials had neither. EMA publication was the earliest information source for 5.8% of pivotal trials, available a median 523 days (IQR 363–882 days) before the earliest publication.

Conclusions The EMA Clinical Data website contains lengthy clinical trial documents. Almost half of submissions to the EMA were based on single pivotal trials, many of which were phase 1 trials. CSRs were the only source and a timelier source of information for many trials. Access to unpublished trial information should be open and timely to support decision-making for patients.

  • Clinical trials
  • PRIMARY CARE
  • Quality in health care

Data availability statement

Data are available in a public, open access repository. All data used in this study are publicly available from the EMA at: https://clinicaldata.ema.europa.eu/web/cdp/home. Extracted data from this study are available at: https://doi.org/10.5281/zenodo.7700338.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • This study characterises and describes the entire selection of clinical study reports made publicly available by the European Medicines Agency (EMA).

  • We use clinical study report (CSR) filenames to identify and navigate individual drug trials and document types.

  • This study examines the timeliness of publication of results of pivotal trials submitted to the EMA for regulatory drug approval, and compares the availability of their results across CSRs, journals and clinical trial registries.

  • Given the large volume of documents and trials, completing a more thorough analysis of the characteristics and contents within the available CSRs was not feasible.

  • While the methods for identifying matched publications were rigorous and independently checked by three authors, some publications may not have been identified.

Introduction

Background and rationale

There is a growing body of evidence related to reporting biases in clinical trials which has been highlighted across a wide variety of treatments including antidepressant and oncological medications.1–7 Published studies of clinical trials are biased towards positive results.5 8 Considering only published studies has been shown to increase the apparent clinical effect size by one-third.5

Clinical study reports (CSRs) and other documents used to obtain regulatory approval for medications provide a rich and authoritative description of clinical trials, including details on study design, methods and results which may be absent or inaccurate in published papers.9–13 CSRs have been an increasing focus in research.14 The benefits of using such unpublished information in evidence synthesis and the subsequent impact on clinical efficacy estimates, clinical recommendations and decision-making have been demonstrated.10 13 15–17 Even when access to this more detailed information on clinical trials is possible, challenges have been highlighted, including inefficiencies in navigating vast amounts of documents and incomplete and delayed access to the information.18 There are ongoing calls for prompt, unrestricted access to CSRs to allow for a more robust appraisal of trials of new medications.18–21

EMA clinical trial publication and request policies

Since 2010, the EMA has released clinical trial reports, which include CSRs, on request under its ‘access to documents’ policy (EMA policy 0043),22 however, requesting CSRs by this method has been shown to be a long process with lengthy delays.23

On 1 January 2015 the EMA introduced the first phase of a new policy involving the publication of clinical data for newly approved medications, entitled ‘EMA Policy 0070’.14 24 25 This policy requires that approval documents submitted to the EMA as part of a Marketing Authorisation Application (MAA) by a drug company, be made publicly available for non-commercial purposes.26 The first documents were published in October 2016 on the EMA’s Clinical Data website,27 however, in December 2018, the EMA temporarily suspended the automatic publication of these regulatory documents, citing a need to prioritise implementation of their Business Continuation Plan, a contingency plan which relates to the UK’s withdrawal from the European Union (EU). To date this process remains suspended due to the impact of the COVID-19 pandemic,28 with the exception being an agreement to publish clinical trial information on COVID-19 medicines, including vaccinations.29

Pivotal CSRs and drug approval pathways

It is generally recommended that drug approvals should be based on at least two clinical trials, often described as pivotal trials.30 31 The exact definition of a pivotal trial however is unclear, with a paucity of information available32 and no clear consensus between regulators.33 The increasing number of drug approvals based on a single pivotal trial, particularly for orphan drug designation, has been highlighted as a concern with the preference being for a more rigorous evidence base.33 34 CSRs of trials submitted to the EMA are labelled as pivotal during the submission process as advised by the regulator.35 Characteristics of the pivotal studies available on the EMA’s Clinical Data website have yet to be summarised, with similar information previously described for such submissions to the Food and Drug Administration (FDA).36

Publication of trial results

Despite the importance of accessing clinical trial results promptly, delays in the publication of clinical reports by the EMA under Policy 0070 have recently been highlighted.12 37 Paludan-Müller et al used the published information available from the EMA resource to describe the characteristics of the available trials, specifically the delay in publication of trial information relative to the EMA’s planned timelines.37 They found that for only two medications, the EMA published the information within the recommended timelines, with a median delay of 446 days.

Overall, the EMA Clinical Data website represents a significant source of evidence for research purposes. Greater access to unpublished clinical trial information would allow for such rich sources of information to be used more widely in research, including evidence synthesis and decision making. Several studies have described the characteristics of unpublished trials using non-random samples of CSRs.9 12 To date, as far as the authors are aware, a comparison of the availability of clinical trial results from the EMA and from conventional published sources, including clinical trial registries and journal publication, has yet to be undertaken. This is relevant for those researchers hoping to include such additional trial information in their analyses, including evidence synthesis.

Objectives

The objectives of this study are:

  • To describe the characteristics of the available CSRs and related regulatory documents, which have been published by the EMA on its Clinical Data website since 2016, summarised by medication characteristics, trial characteristics and document characteristics.

  • To describe the characteristics of those trials labelled as ‘pivotal’, specifically document length and trial phase.

  • To report the availability and timeliness of access to results from these pivotal trials across three sources; CSRs, trial registries and journal publications.

Methods

Study design

This is a cross-sectional study of all documents submitted to the EMA by a Marketing Authorisation Applicant for regulatory drug approval, published the EMA. This study follows the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies and the relevant STROBE checklist is available in online supplemental materials.38

Supplemental material

Data sources and data extraction

CSR documents

Summary information on each medication for all regulatory submissions published by the EMA under policy 0700 were downloaded in comma-separated value format from the EMA’s Clinical Data website. Such information included authorisation type, approval pathway (including specific additional pathways such as submissions under orphan status), procedure type and Anatomical Therapeutic Chemical (ATC) classification code. Individual CSR documents for each of the medications were also downloaded and saved in PDF format.

The pdftools package in ‘R’ statistical software39 40 was used to extract individual CSR document filenames and page length for all documents for each medication. Using the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH)41 structure and EMA guidance35 on the naming of CSR documents, filenames were used to identify individual trials with which the documents were associated. Filenames were screened for substrings which identified individual trials by their trial identification number, the specific document type and whether the trial was classed as pivotal ‘p-CSR’ or supportive ‘s-CSR’ (eg, m5351-(trialid)-p-csr-body, m5351-(trialid)-p-app1612-crf).

For the subset of trials labelled as ‘pivotal’ CSRs, using a Microsoft Excel spreadsheet, further information was extracted from each document on date of completion of the trial’s CSR (the final completion date reported within the CSR) and date of CSR publication by the EMA. Some trials were listed as a combination of phase 1, 2 and 3 trials (eg, phase 1b/2, phase 2/3), however, in these cases for the purposes of analysis in this study, the higher of the two phases was used.

Journal publications and trial registries

For pivotal studies with an included CSR, literature databases (Medline (PubMed), Embase and Google Scholar) were searched for the primary journal publication for each trial. Searches were conducted initially using the trial ID. If no publication was found, searches were conducted using the ClinicalTrials.gov or European Union Drug Regulating Authorities Clinical Trials Database (EudraCT) trial registry numbers or a combination of the active drug substance and the number of randomised trial participants. The single earliest primary publication reporting results related to the individual trial was used where available. Pooled analyses were not included. For trials with no primary publication found, the first publication which included trial results was used (including where results were reported along with those from other related trials). For CSRs which were reports of interim analyses, the publication with the closest length of follow-up to the CSR was chosen. Length of journal publication and first publication date were extracted, using the main body of the publication excluding appendices.

Similarly, the clinical trial registry Clinicaltrials.gov was searched to identify entries corresponding to trials. Initially, the registry was searched using trial ID number and if not found, then searches were conducted using drug substance, filtered by phase, with the registrations checked by matching criteria including trial dates and number of participants. If no registration was found then the WHO International Clinical Trials Registry Platform was searched using the trial ID number. The presence of a corresponding journal publication and/or trial results on clinical trial registries were documented, along with relevant publication date for these stages and the number of pages of the main text for the trial publication.

Initial searches of registry and publication sources were conducted by one author (DB). A second author (CP) verified the extracted information and repeated the search for trials where no registry entry or publication were identified. Outstanding trials at that stage were searched for by a third author (FM). Discrepancies were discussed between authors and resolved by consensus. All data analysed in our study are available in the Zenodo repository.42

Statistical methods

Using Stata software V.15,43 descriptive statistical analysis was carried out on the extracted data at the medication, trial and document levels, as well as for the pivotal trials. Median and IQRs as well as mean and SD were reported for relevant outcomes.

Analysis of medication, trial and document characteristics

All medications for which information were available were included in the initial overall descriptive characteristics. Medications were then excluded from further analysis if there were no documents provided, if none of the documents provided were related to individual trials, or if the documentation for a medication entirely duplicated those of another medication. These were then excluded from further analysis in order to obtain more accurate characteristics at the trial and document levels.

The number of documents, total number of pages and number of trials per medication were calculated, as well as the total number of pivotal and supportive trials and the total number of pages per document type. Differences in number of trials and documents per medication, by medication characteristics were assessed using the Wilcoxon rank-sum test (Mann-Whitney U) for two-sample comparisons and Kruskal-Wallis test for multi-sample comparisons.

Negative binomial regression analyses were used to investigate the association between medication characteristics and the number of pivotal trials per medication. This model was selected over a Poisson model due to over-dispersion in the variance compared with the mean count of pivotal trials. Univariate negative binomial regression analyses were initially conducted, followed by multivariable analysis including all factors, generating incident rate ratios (IRR) and 95% CIs. Medication characteristics included were procedure type, approval pathway, generic status and therapeutic area (based on ATC code). These were chosen as they had shown significant associations with number of overall trials in previous non-parametric analyses.

The classification of each document by module type was extracted. A module is a component of a regulatory submission as outlined by the ICH. In addition, the number and proportion of trials which included separate documents for CSR body, statistical analysis plan (SAP), case report form (CRF) and protocol were reported, as well as those which included all four document types. At the document level, the total number of pages available from the EMA, as well as the number of pages per document type were reported.

Inclusion process for pivotal trials

Any pivotal trials which were also included in another submissions were excluded as duplicate trials. Non-randomised trials were also excluded.

Descriptive statistics of pivotal trials

For the subgroup of pivotal trials, the trial phase reported in the CSR was summarised. The proportion of all medications which were supported by a single pivotal trial was calculated and also the proportion of these studies by trial phase.

Publication availability and timeliness

The availability and timeliness of access to the results of these pivotal trials were also described, by calculating the length of time in days between the documented date of CSR completion, the date of release of documents by the EMA (herein defined as ‘EMA publication’) and publication of the clinical trial results in a journal publication or clinical trial registry. For those trials with both a journal publication and with clinical trial registry results, the earlier of the two dates was used. These time periods were plotted using both time from EMA publication and CSR completion dates as reference points.

CSR and journal publication length

We also calculated the ‘compression factor’ of included trials, which is a ratio of the page length of the pivotal CSR documents to that of the matched journal publications. This measure has been used previously to represent the compression and loss of information between a CSR and a typical journal publication.9 For trials whose CSRs bodies were in multiple parts, the total number of pages for all parts were used. For trials whose results were published in journals along with results of related trials for the same medication, the publication length was divided by the number of trials included in the paper.

Public and patient involvement

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

Results

Descriptive statistics of medication, trial and document characteristics

Medication characteristics

In total, there were regulatory drug approval submissions for 142 medications published on the EMA’s Clinical Data website between 2016 and 2018. The characteristics of these medications are shown in table 1. The majority of applications for these medications were for initial marketing authorisations and authorised products. There were 45 approvals via alternative or special pathways, the largest proportion of these (57.8%) were medications under orphan designation, which comprised 18.3% of all medications. By ATC classification, antineoplastic and immunomodulating agents accounted for the largest proportion (40.9%) of medications.

Table 1

Characteristics of all medications (n=142)

The process for including medications in further analysis is demonstrated in figure 1. Out of the 142 submissions, 14 were excluded as there were no documents provided. Four medications were excluded as no documents relating to trials were provided. Of these, the documents provided included only information on systematic literature reviews, meta-analyses, pooled analyses and immunogenicity reports and for one medication all available documents were exemption notices. There were five medications which were excluded as duplicate submissions, in that identical CSR documents were published for two product submissions, with a different product name used for each. After these exclusions, there were 119 unique medication submissions included in the further analysis. Table 2 summarises the trials and documents at the medication level. There was a median of 15 (IQR 5–46) documents, 5 (IQR 2–14) trials and 9629 (IQR 2711–26 673) pages per medication submission for drug approval. The median number of pivotal trials per medication was 2 (IQR 1–3). For 46.2% (55/119) of medications there was just a single pivotal trial submitted for regulatory drug approval. In contrast, one medication had 17 pivotal trials included in its submission.

Table 2

Characteristics of medications included in further analyses (n=119)

Figure 1

Inclusion and exclusion of medications and trials in analyses. CSRs, clinical study reports; EMA, European Medicines Agency; pCSR, pivotal CSR.

There was a wide variation in the numbers of trials and pages per medication when stratified by different approval pathways and therapeutic areas (online supplemental table S1). Following Wilcoxon rank-sum and Kruskal-Wallis tests for comparisons, there was a significant difference in the median number of trials per medication for procedure types and conditional approval pathways. There was also a significant difference in the median number of pages per medication for conditional approval and for therapeutic class (online supplemental table S1).

In regression analysis of the number of pivotal trials per medication, there was an increased number of pivotal trials associated with initial marketing authorisation (IRR 2.09 (95% CI 1.48 to 2.95)) compared with extension of indication. There was an increased number of pivotal trials associated with anti-infective therapeutic class (IRR 1.89 (95% CI 1.31 to 2.75)) compared with the therapeutic class reference category (antineoplastic and immunomodulating agents). Generic status was significantly associated with a lower number of pivotal trials per medication (IRR 0.31 (95% CI 0.18 to 0.55)). Results of regression analyses are shown in table 3.

Table 3

Results of univariate and multivariable negative binomial regression models of medication characteristics associated with number of pivotal trials per medication

Trial characteristics

In total, 2115 trials were included in the analysis. There were 1678 (79.3%) studies identified as supportive (sCSR) and 264 (12.5%) as pivotal (pCSR). The remaining 173 studies (8.2%) were neither tagged as supportive nor pivotal and they mostly comprised in-vitro studies and pharmacokinetic reports. There was a median of 1 (IQR 1–4) document per trial and a median of 336 (IQR 21–1192) total number of pages per trial. There was a significant difference in the median number of pages per trial for procedure types, orphan and biosimilar approvals and for therapeutic class (online supplemental table S2).

Document characteristics

Across all 142 drug regulatory submissions, there were 6317 documents published by the EMA with a total of 3 155 354 pages. Online supplemental table S3 summarises these documents by module type. There were 16 instances in which the protocol, SAP and CRF were included in one document and were therefore identified as individual documents in our analysis (online supplemental figure S1) meaning there were 6301 individual identified documents included in this analysis. Of these, 2465 (39.1%) were CSR body documents, 1182 (18.8%) were protocols, 840 (13.3%) were SAPs, 794 (12.6%) were sample CRFs, 118 (1.9%) were integrated summaries of safety, 53 (0.8%) were integrated summaries of efficacy and 849 (13.5%) were other documents (online supplemental figure S1). The median number of pages per document was 92 (IQR 23–321) which varied by document types (table 4).

Table 4

Length of documents by document type

Of 2115 unique trials, the majority, 1945 (91.9%), included separate CSR body documents, however just 757 (35.8%) included all four of CSR body, CRF, SAP and protocol documents separately (online supplemental table S2).

Descriptive statistics of pivotal trials

Of the 119 unique submissions to the EMA for drug approval, there were a total 264 trials labelled as pivotal. Following exclusions of duplicates and of one trial which was excluded as it was not a randomised trial, there were 243 pivotal trials included in the analysis (figure 1).

Of these 243 pivotal trials, there were 148 (60.9%) phase 3 trials, 48 (19.8%) phase 2 trials, 45 (18.5%) phase 1 trials and two (0.8%) phase 4 trials (figure 2, online supplemental figures S2 and S3, table S4). Of the 119 included submissions, there were 55 medications (46.2%) for which their submission was supported by a single pivotal trial. One submission was based on a phase 4 trial, 31 were based on a single phase 3 trial and 6 were based on a single phase 2 trial. Sixteen submissions were based on a single pivotal phase 1 trial, which amounted to 13.4% of all 119 approvals. Of the 16 trials, 13 were related to initial marketing authorisation and 12 were for generic medications (online supplemental figure S4). Fifteen (93.8%) of these submissions based on a single pivotal phase 1 trial had no other supportive trials included in the application; one submission did include two further supportive phase 3 trials.

Figure 2

Included pivotal trials by phase and availability. EMA, European Medicines Agency.

Figure 3

Plot of journal/registry availability from time of EMA publicaton (above) and CSR completion (below). CSR, clinical study report; EMA, European Medicines Agency.

Publication availability and timeliness

Of the 243 pivotal trials, results were not identified on ClinicalTrials.gov for 68 (28.0%) trials, with four of these found on other clinical trials registries; three on the EU Clinical Trials Register and one on the Japanese Primary Registries Network. For 64 trials (26.3%), no trial registry entries or results were found. Journal publications were not located for 41 (16.9%) trials, of these 33 (80.5%) were phase 1, 2 (4.9%) were phase 2 and 6 (14.6%) were phase 3 (figure 2, online supplemental table S4). There were 33 trials (13.6%) for which neither clinical trial registry results, nor journal publications were available.

For 94.2% (195/207) of pivotal trial CSRs with a matched source, results were available in a publication or registry prior to EMA publication. However, for 5.8% (12/207) of matched pivotal trials, the EMA publication was the earliest source of results, available a median of 523 days (IQR 363–882 days) before the earliest corresponding publication in a journal or clinical trial registry (figure 3, table 5).

Table 5

Time difference and CSR delays (days)

When dates of CSR completion were compared with matched published sources, CSRs were found to have been completed before journal or trial registry publication in 176 (85.0%) pivotal trials. These CSRs had been completed a median of 338 days (IQR 173–554 days, range 10–2367 days) before the date of journal or registry source publication (figure 3, table 5).

CSR and journal publication length

The median document length for pivotal CSR body documents was 1392 (IQR 545–3099) pages with a mean of 2426.5 (SD 3 016.8) pages. For journal trial publications, the median length was 10 (IQR 9–12) pages with a mean of 10.6 (SD 3.2) pages per publication. The compression factor ranged from 9.7 to 3361, and the median was 164.2 (IQR 76.3–316.7), with a mean of 277.4 (SD 358.2). Median compression factor was larger for extension of indication and was smaller for trials for medications under exceptional circumstances (online supplemental table S5).

Discussion

Key results

There was a median of 15 (IQR 5–46) documents, 5 (IQR 2–14) trials and 9629 (IQR 2711–26,673) pages per submission for drug approval published by the EMA between 2016 and 2018. Out of 119 unique drug regulatory approval submissions, 46.2% were supported by a single pivotal trial and 13.4% were based on a single pivotal phase 1 trial, which were mostly for generic mediations. There were 33 trials (13.6%) for which neither clinical trial registry results, nor journal publications were available. For 5.8% of pivotal trials, the EMA publication was the earliest information source for trials, with a median time difference of 523 (IQR 363–882) days between EMA and earliest journal or registry publication. CSRs were found to have been completed before journal or trial registry publication in 85.0% of pivotal trials and completed a median of 338 days (IQR 173–554 days, range 10–2367 days) before the date of journal or registry publication.

Interpretation and comparison with existing literature

We report a median of 15 (IQR 5–46) documents and 5 (IQR 2–14) trials per medication, which are similar to results reported by Paludan-Müller et al.37 However, we included all available study types in our analysis and reported 2115 unique trials, compared with 1005 reported by Paludan-Müller et al who did not include smaller clinical studies.

This study reports a median document length of 92 (IQR 23–321, range 1–42,084) pages across all documents and 181 (IQR 21–679) pages for CSR body documents. These estimates are shorter compared with those in a similar study by Doshi and Jefferson which reported a median document length of 644 pages (range 9–15 440).9 This may be due to the fact that Doshi and Jefferson used a non-random sample of CSRs requested from various sources. In addition, our study used all available trials, including supportive, phase 1 and undesignated trials which were shorter documents.

We reported an overall median compression factor of 164.2 (IQR 76.3–316.7, range 9–3361) using a sample of just pivotal trials for this estimation. We also report how such a measure varies between types of approval pathway. Doshi and Jefferson report a median compression factor range of 1–1221 for a non-random sample of CSRs, including all trial types, including supportive and undesignated trials. Schroll et al reported a compression factor of 71–270 in a sample which included just module 1 and 2 documents.44 These results highlight that compression factor in itself is a crude measure of the degree of condensing of trial information.

To our knowledge, this is the first study which summarises characteristics of all pivotal trials for which the EMA have published CSRs and related documentation since 2016. This study reports a median of 2 (IQR 1–3) pivotal trials per regulatory drug approval submitted to the EMA, which is similar to results published by Dhodapkar et al who report a median of 1 (IQR 1–2) pivotal trial per approval submission to the FDA between 2017 and 2019.30

The EMA would consider a clinical trial as pivotal if it provides ‘robust confirmatory evidence of a clinically relevant effect in the target population with an acceptable safety profile, taking available therapies into account’.32 Where submissions for approval are based on a single pivotal trial, the EMA advise that the study should be ‘exceptionally compelling’ with a particularly high standard of trial design with respect to internal and external validity, clinical relevance, statistical significance, data quality and internal consistency.34 However, in this study, almost one in five studies (18.5%) identified as pivotal trials were phase 1 studies. Our results show that 46.2% of all EMA regulatory approvals between 2016 and 2018 were based on a single pivotal trial. These align with results published by Morant and Vestergaard who reported that 45% of newly approved medications by the EMA between 2012 and 2016 were based on a single pivotal trial33 and Downing et al who report a figure of 37% for approvals by the FDA between 2005 and 2012.31

In addition, in our study, of those submissions supported by single trials which were highlighted as pivotal, almost one-third (29.6%) were phase 1 trials, meaning that 13.4% of all submissions to the EMA were based on a single pivotal phase 1 trial. This represents an increase in the proportion of approvals based solely on uncontrolled trials. Hatswell et al reported that between 1999 and 2014, 44 out of 795 (5%) approvals issued by the EMA were based on uncontrolled trials.45 While the concept of pivotal bioequivalence trials is described,46 this concept would be at odds with the more conventional definitions of pivotal trials6 32 and also seem to conflict with the EMA’s own guidance on pivotal trials that they should provide compelling evidence of efficacy and safety for the medications in question.34

Our results show that for 16.9% of pivotal trials, no journal publication was found and for 13.6%, neither clinical trial registry results nor journal publications were available. While many of these were for phase 1 trials, it may be reasonable to expect that if a trial was defined as pivotal, such trial results should be available from conventional published sources. In addition, there were some phase 3 trials for which CSRs were the only available information source.

Our results also show that for 5.8% of pivotal trials, publication of CSRs by the EMA was the earliest source of trial information, available a median of 523 days (1.43 years) before the earliest publication or trial registry results. Paludan-Müller et al noted a median delay of 1.21 years between EMA publication of CSRs compared with the timeline stated under policy 0070, and so had CSRs been published on schedule, an even higher proportion may have been the earliest source of trial results.12

We also showed a large time delay between CSR completion and availability of results from publication, trial registry or even the EMA’s Clinical Data website, with a median time difference of 338 days (IQR 173–554). A recent study by Lythgoe et al reported similar results, showing a median time to journal publication of 344 (IQR 149–544) days, however, this was from the date of oncological drug approval by the EMA as opposed to CSR completion.47 We reported overall delays of up to 2367 days. The EMA do report targets within which they aim to make CSRs available.28 Even accounting for these timelines, for many CSRs it would seem that there is still a sizeable delay in availability of results from either the EMA or Marketing Authorisation Holder following the approval process. However, the reasons for these delays are uncertain, potentially due to delays in the MAA submission to the EMA and delays in the EMA reviewing the submitted information and making a decision. We propose that earlier public access to such documents, for example, to clinical researchers and decision makers, represents a further potential opportunity improve access to clinical trial information.

Study strengths

This is one of the first studies which characterises and describes the entire selection of CSRs made publicly available by the EMA. We use a novel method of using CSR filenames to identify and navigate individual drug trials and document types. This allowed for a more detailed description of all included trials and documents, and specifically those trials identified as pivotal by the marketing authorisation applicant. An additional strength was our use of a systematic approach for searching and identifying corresponding published sources of trial results in journals and trial registries, in addition to considering the timeliness and availability of such information compared with the EMA. This study highlights the potential opportunity for earlier access to such information on clinical trials. As far as we are aware, this study is the first to examine the timeliness of publication of results of pivotal trials submitted to the EMA for regulatory drug approval, and to compare the availability of their results across CSRs, journals and clinical trial registries.

Study limitations

Given the large volume of documents and trials, completing a more thorough analysis of the characteristics and contents within the available CSRs was not feasible. Presence of individual CSR sections, such as CSR body, protocols, SAPs and CRFs within the documents were not explored. While the methods for identifying matched publications were rigorous and independently checked by three authors, some publications may not have been identified. Publication length, and hence compression factor, is a crude measure of the amount of information contained and lost across each source, and therefore does not provide an indication of what important information may be absent from publications relative to CSRs.

Conclusions and recommendations

Information on submissions for marketing authorisation approvals published by the EMA on its Clinical Data website includes large number of documents, however, the ICH naming structure can be used to identify individual trial and document types to navigate them more efficiently. We described the wide range of numbers of documents, pages and trials per approval application for each medication. We showed that most submissions included individual documents for the common recommended CSR document types (body, CRF, SAP and protocol), however, just one-third published all documents individually. EMA Policy 0700 mandates that these four specific sections, should be available for relevant trials.11 35 48–52 Such clinical trial information publicly available on the EMA Clinical Data website should be consistent across medications and trials. Absence of individual documents could be a potential barrier to full transparency and appraisal of information about particular trials. The marketing authorisation applicant should endeavour to include each of these document types individually in their submissions and the EMA should encourage this practice.

The definition of a ‘pivotal’ trial appears ambiguous, and there seems to be a disparity between those trials defined as pivotal for regulatory purposes and those which are compelling sources of evidence of efficacy and safety, with many non-randomised or preclinical trials being defined as such. More clarity on what constitutes a pivotal trial from drug regulators and adoption of a more uniform definition would be beneficial.

Researchers should consider using CSRs in evidence synthesis as they are often a timely and sometimes only source of information for trials. Search strategies for systematic reviewing should include strategies for identifying CSRs from relevant sources, including the EMA’s Clinical Data website. The standardised ICH naming structure could be used by, for example, researchers to more efficiently navigate submissions and identify relevant documents.

We have shown the lack of timeliness of access to CSRs for some trials, which can be a missed opportunity and barrier to their use in research. We would recommend the resumption of publication of CSRs by the EMA under policy 0070 as an important measure to improve transparency, and with more timely publication of CSRs. The Marketing Authorisation Applicant or the EMA also could potentially provide access to CSRs sooner following their completion to further enhance the availability of such evidence. Researchers and policy makers should advocate for unrestricted access to unpublished trial information, to support timely clinical decision-making.

Data availability statement

Data are available in a public, open access repository. All data used in this study are publicly available from the EMA at: https://clinicaldata.ema.europa.eu/web/cdp/home. Extracted data from this study are available at: https://doi.org/10.5281/zenodo.7700338.

Ethics statements

Patient consent for publication

Ethics approval

Not applicable.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @DaveByrneDB, @FrankMoriarty

  • Contributors DB, FM and TF all contributed to the conception of this study. DB was involved in conceptualisation, data curation, formal analysis, investigation, methodology, project administration, writing of original draft and review & editing. CP was involved in data curation, formal analysis and draft review & editing. TF was involved in conceptualisation, funding acquisition, project administration, supervision, writing of original draft and review & editing. FM was involved in conceptualisation, funding acquisition, project administration, supervision, data curation, formal analysis, investigation, methodology, writing of original draft and review & editing. DB drafted and revised the manuscript and all authors were involved in the approval of the final manuscript. FM is the guarantor.

  • Funding This doctoral research project is funded by the Health Research Board of Ireland as part of the RightCare Collaborative Doctoral Award (funding reference CDA-2018-005). The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.