Article Text
PDF
Abstract
Introduction A review of European Union (EU)-funded initiatives linked to ‘Real World Evidence’ (RWE) was performed to determine whether their outputs could be used for the generation of real-world data able to support the European Medicines Agency (EMA)’s regulatory decision-making on medicines.
Method The initiatives were identified from publicly available websites. Their topics were categorised into five areas: ‘Data source’, ‘Methodology’, ‘Governance model’, ‘Analytical model’ and ‘Infrastructure’. To assess their immediate relevance for medicines evaluation, their therapeutic areas were compared with the products recommended for EU approval in 2016 and those included in the EMA pharmaceutical business pipeline.
Results Of 171 originally identified EU-funded initiatives, 65 were selected based on their primary and secondary objectives (35 ‘Data source’ initiatives, 15 ‘Methodology’, 10 ‘Governance model’, 17 ‘Analytical model’ and 25 ‘Infrastructure’). These 65 initiatives received over 734 million Euros of public funding. At the time of evaluation, the published outputs of the 40 completed initiatives did not always match their original objectives. Overall, public information was limited, data access was not explicit and their sustainability was unclear. The topics matched 8 of 14 therapeutic areas of the products recommended for approval in 2016 and 8 of 15 therapeutic areas in the 2017–2019 pharmaceutical business pipeline. Haematology, gastroenterology or cardiovascular systems were poorly represented.
Conclusions This landscape of EU-funded initiatives linked to RWE which started before 31 December 2016 highlighted that the immediate utilisation of their outputs to support regulatory decision-making is limited, often due to insufficient available information and to discrepancies between outputs and objectives. Furthermore, the restricted sustainability of the initiatives impacts on their downstream utility. Multiple projects focussing on the same therapeutic areas increase the likelihood of duplication of both efforts and resources. These issues contribute to gaps in generating RWE for medicines and diminish returns on the public funds invested.
- EU-funded initiatives
- real world evidence
- data sources
- methodologies
- governance models
- analytical models
- infrastructure
- sustainability
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- EU-funded initiatives
- real world evidence
- data sources
- methodologies
- governance models
- analytical models
- infrastructure
- sustainability
Strengths and limitations of this study
This is the first comprehensive evaluation of European Union-funded initiatives linked to ‘Real World Evidence’ that looks at their usefulness to support regulatory decision-making on medicines.
The analysis is based on review of the publicly available information provided by each initiative on their English language websites.
The internet search was based on a list of internally agreed keywords which might not be fully exhaustive.
The follow-up period to perform the second and third steps of our analysis was 6 months (January to June 2017). This might not have been long enough to cover the time lag between the finalisation of some of the initiatives and the publication of their final reports, and therefore our analysis may not have taken into account their final outputs.
The website of each initiative was reviewed by individual European Medicines Agency staff members. However, in some cases, there were (1) limited information published on the websites, (2) broken links and (3) both limited access/limited information on access to data, which made it difficult to determine the appropriateness of the initiatives’ attributes for inclusion in the inventory, and conclude on their general applicability to regulatory science.
Introduction
The clinical evidence collected for the marketing authorisation of new medicines traditionally comes from randomised clinical trials (RCTs) but it is recognised that RCT data have limitations including tightly controlled conditions of clinical care, highly selected populations and, in some scenarios, small sample sizes.1 As a result, their applicability to the safety and efficacy of medicines in postauthorisation use is unknown. There is therefore a need to supplement RCTs with other sources more representative of everyday ‘real world’ medical practice in order to provide additional insight on the benefit–risk balance.
According to the GetReal Glossary of Definitions of Common Terms,2 ‘Real World Evidence’ (RWE) derives from the analysis and/or synthesis of ‘Real World Data’ (RWD) that can either be primary data collected in a manner which reflects how interventions would be used in routine clinical practice or secondary data derived from routinely collected data. The range of RWD is wide and sources include electronic healthcare records, patient/disease registries, hospital records and health insurance data/claims databases. The EU Network Strategy to 2020 identifies RWE as a key enabler to bring innovative products to patients with unmet medical needs and to support the safe and effective use of medicines.3
While the European Medicines Agency (EMA) already uses RWE sources in its evaluations, this is typically on an ‘ad hoc’ basis. There is therefore a need to systematically understand RWE outputs at an European Union (EU)-wide level in order to make best use of existing information and to identify areas where further efforts are needed. We created an inventory of EU-funded initiatives linked to RWE and RWD. The objectives were: (1) to identify the initiatives established in terms of RWD sources, relevant methodologies, governance models, analytical models and infrastructures created to facilitate RWD collection, transformation, sharing and analysis; (2) to understand if and how these initiatives could be exploited to support regulatory decision-making on medicines4 ; and (3) to consider the initiatives strategically in terms of needs, gaps and opportunities for the generation of RWD related to the therapeutic areas of medicinal products in the pharmaceutical development pipeline.
Methods
Selection (first step)
A three-step approach was followed (figure 1), starting with an internet search performed by one reviewer (KP) to identify completed and ongoing RWE initiatives funded through the sixth and seventh Framework Programmes (FP6/FP7),5 6 Horizon 2020 (H2020)7 (including the Innovative Medicines Initiative (IMI)8), other EU initiatives (eg, European Research Council, Strengthening Collaboration for Operating Pharmacovigilance in Europe Joint Action,9 HIMSS Europe Governing Council10) and nationally funded initiatives discovered during the search. The search terms used were: registry/ies, real world evidence, clinical, electronic medical record, eHealth, big data, electronic health record, data linkage/link, paediatrics, pregnancy, geriatrics, hospital data, data source, unique patient identifier, coding terminology, governance model, common protocol, distributed data approach, pooling of data and analytical model. The search cut-off date was 31 December 2016; as such, all initiatives identified via the search terms which began before or on that date and were funded by either the EU or national bodies were included.
Development steps of the inventory of European Union-funded ‘Real World Evidence’ initiatives.
Initiatives started on or before this date that matched any of the keywords were included in an Excel tracking table (see online supplementary file, section 1.1). Based on their website information, they were categorised according to the development of potentially relevant (1) RWD sources, (2) methodologies, (3) governance models, (4) analytical models or (5) infrastructure that could help provide evidence to support regulatory decision-making (see online supplementary file, section 1.2.1). Initiatives could fall into more than one category. Those that did not focus on one or more of the categories or that focused only on the early stage of drug development (eg, preclinical or clinical trials, development of molecular compounds/biomarkers), on the ‘-omics’ (eg, genomics) or on diseases with limited geographical spread (eg, Ebola outbreak) (figure 2) were considered out of scope of the inventory and were therefore not selected for further analysis.
Supplementary file 1
Selections of initiatives for the inventory of European Union-funded ‘Real World Evidence’ initiatives.
Category attributes (second step)
For each of the five categories, attributes were defined to facilitate selection of the outputs potentially most relevant for assisting in regulatory evaluations (eg for data sources, the attributes included access, database linkage, unique patent identifier or paediatrics data). Three reviewers (KP, PM, AP) with experience in pharmacovigilance and regulatory science identified the primary and secondary objectives of the initiatives, their attributes and their corresponding deliverables through examination of the initiatives’ webpages and review of published documents including reports, presentations and publications. They then assigned the initiatives to one or more categories as appropriate (see online supplementary file, sections 1.1 and 1.2.2). In cases of doubt, the three reviewers met face-to-face to achieve consensus.
Scoring (third step)
A dashboard listing all the attributes was created for each category. Each initiative was categorised and characterised through identification and scoring of its attributes. Scores ranged from 0 to 2 for ‘Yes/No’ questions, for example, on whether the data source included paediatric data (where 0=‘No’, 1=‘Unknown’ and 2=‘Yes’), or from 0 to 3 for more qualitative questions, for example, on the type of care covered (0=‘Unknown’, 1=‘Primary’, 2=‘Secondary/Tertiary’ and 3=‘Primary, Secondary, Tertiary’).
Two attributes were scored across all dashboards: (1) the availability of published outputs at the end of each completed project, and (2) whether the published outputs fully addressed the originally stated objectives. The dashboards permitted the creation of graphs for each of the categories showing the number of initiatives fulfilling each attribute according to the information provided on their websites.
Initiative partners and funding
The numbers of partners named in each initiative and the amount of funding awarded were included in each dashboard.
Therapeutic areas of the initiatives, new medicinal products and EMA business pipeline
The therapeutic areas of the initiatives included in the data source category were mapped to:
Those of the products recommended for approval through the EU centralised authorisation procedure by the Committee for Medicinal Products for Human Use11 in 2016.
Those of the products included in the EMA pharmaceutical business pipeline12 through which developers have expressed a clear intent to submit a marketing authorisation application (MAA) through the centralised procedure to the EMA between March 2017 and December 2019.
Patient involvement statement
This descriptive analysis did not involve any patients.
Results
The first screening returned 171 potentially relevant initiatives (see online supplementary file, section 1.1) that matched at least one of the search keywords. Following the second screening, 115 initiatives were reviewed for categorisation (51 FP, 30 IMI, 15 H2020, 12 other EU initiatives, 7 nationally funded initiatives), of which 65 fell into one or more of the five categories (figure 2). Some initiatives were included in more than one category, for example, data source initiatives that also developed infrastructure. The selected initiatives had a median duration of 5 years.
Dashboards scoring and summary of findings
Initiatives were scored in the dashboards according to their category(ies) attributes. The main findings for each category are summarised below and illustrated in figure 3. The scoring dashboards are provided in the online supplementary file, section 2.1.
Attributes of the European Union (EU)-funded initiatives according to each of the five categories. EHR, electronic health record; RW, real world.
Data sources: Information on access to the data sources developed through the initiative was available on the websites of 5 of the 35 data source initiatives, including two where only the data custodian had access, and three where access was open to collaborators. Access information was either not publicly available or unclear for the remaining 30 initiatives.
Methodologies: 15 initiatives developed methodologies that could be applied to RWD studies, including guidance on protocol design (9), on the management of bias/confounders (5) and on the use of electronic health records and/or registry data (6). Seven initiatives referred to the use of established diseases or drug coding terminologies such as MedDRA, Snomed CT, ATC or Orphanet.
Governance models: 10 initiatives dealt with confidentiality and data protection aspects, while six related to a new code of conduct.
Analytical models: 10 of 17 initiatives aimed to develop smart text analysis tools such as machine learning, natural language programming or data mining. Data transformation (anonymisation, privacy and compression) was referred to in nine initiatives.
Infrastructure: All 25 initiatives related to the development of platforms or websites to share, extract and store data, of which two also mentioned cloud-based technologies. Three developed a smartphone application that could be used by stakeholders including patients to record personal health information or report suspected adverse drug reactions.
Outputs versus objectives
The outputs matched the objectives for all 5 of the completed ‘Governance model’ initiatives (100%), 5 of the 6 completed ‘Methodology’ initiatives (83%), 10 of the 12 completed ‘Infrastructure’ initiatives (83%), 15 of the 23 completed ‘Data source’ initiatives (65%) and for 7 of the 12 completed ‘Analytical model’ initiatives (58%) (table 1).
Number of completed initiatives with outputs matching objectives
Initiatives’ sustainability
While some initiatives continued beyond their agreed timelines through new EU-funded programmes within, for example, H2020, or were followed up by existing networks (eg, foundations, associations), others ceased at the end of their allocated time and budget without a sustainability plan for any of the outputs delivered. Of the 23 completed ‘Data source’ initiatives, 7 had a clear sustainability mechanism described on their website, for example, the European Medical Information Framework Project.13 Some more recent initiatives integrate roadmaps for future development and continuation as part of their deliverables, like RETHINK big.14
Comparison of therapeutic areas
Of the 14 different therapeutic areas where products were recommended by the EMA for approval in 2016 and of the 15 represented in the EMA business pipeline, 8 were studied by the initiatives (figures 4 and 5). These included oncology (five initiatives), neurology/central nervous system (CNS) (seven), respiratory (five), immunology (five), rheumatology (three) and metabolic diseases (two). Other therapeutic areas where new products were about to be approved or MAAs submitted were not represented in the initiatives, including in particular haematology, gastroenterology or cardiovascular system.
Specific therapeutic areas of products recommended for approval in 2016 versus those covered by the identified initiatives. CNS, central nervous system; RW, real world.
Specific therapeutic areas of products included in the European Medicines Agency business pipeline (March 2017 to December 2019) versus those covered by the identified initiatives. CNS, central nervous system; RW, real world.
Number of partners and funding
Information on funding was available on the websites of 53 of the 65 initiatives categorised and scored. The total funding was 734 million Euros ranging from 1 to 56 million Euros. Information on the numbers of partners was provided for 58 initiatives and ranged from 1 to 161 partners.
For large initiatives (>20 partners), there was no clear relation between the number of partners and the amount of funding (figure 6, funding of the 30 ‘Data source’ initiatives). The same partners were involved in multiple initiatives, especially pharmaceutical companies as well as academic centres.
Number of partners versus funding of ‘Data source’ initiatives.
Discussion
To our knowledge, this is the first evaluation of EU-funded initiatives in terms of their potential to develop RWD sources, methodologies, governance models, analytical models or infrastructure to contribute to the utility of RWE. It highlights discrepancies between the objectives of the identified initiatives and their outputs, a lack of sustainability of the outputs and a mismatch with the therapeutic areas of drugs recently recommended for approval by the EMA and appearing in its business pipeline. Nevertheless, the inventory will be useful for the EMA on receipt of enquiries arising in the course of regulatory assessments of medicinal products as the five dashboards and the relevant attributes may provide sources of additional evidence to support decision-making.
Observation
Of 35 ‘Data source’ initiatives, 27 developed RWD sources that focused on diseases in eight specific therapeutic areas, more than the half of which (17) clustered in neurology/CNS, oncology and respiratory (figures 4 and 5). These multiple parallel initiatives with a substantial number of partners sometimes involved in the same projects may increase the risk of (1) duplication of efforts and resources, (2) overlap of final outputs and (3) generation of unsustainable and non-interoperable data sources. Most initiatives consisted of standalone projects, and while a handful provided a clear set of definitions, most provided no apparent plan to enable the delivery of a sustainable platform on, for example, future data generation, access and reuse. This may result from an omission of the funding bodies to request such a plan or from the lack of incentives, for example, the lack of ongoing funding options.
The inventory highlights that some areas of new product development such as haematology, gastroenterology and cardiovascular system are not represented in the RWD sources identified in the inventory. Therefore, opportunities to generate RWD to support evaluation and decision-making on new products could be created through new initiatives targeting these therapeutic areas. It is possible that such omissions will be addressed to some extent by the strategic research agenda of the recently launched IMI 215 (eg, Big Data for Better Outcomes Programme16).
While the inventory focuses on EU-funded initiatives, the same exercise could be extended to international initiatives in order to enlarge the geographical spread and number of data sources, provide a better understanding of populations studied in real world settings worldwide and widen the possibility of linkage between data sources through new analytical models.
Limitations
The three consecutive steps were based on the review of the English version of the websites by individual EMA staff members. The internet search was based on an agreed list of keywords which might not have been fully exhaustive and therefore some relevant initiatives may be missing. The follow-up period to perform the second and third steps of our analysis was only 6 months (January to June 2017), and therefore does not take into account any final outcomes of the originally identified initiatives that were published afterwards. However, in general, limited information published on the websites (eg, lack of final reports or accounts of deliverables while some initiatives did not even have a website), broken links and both limited access/limited information on access to the data made it sometimes difficult to determine the initiatives’ attributes for inclusion in the inventory, let alone conclude on their general applicability to regulatory science. As the initiatives were publicly funded, it is reasonable to expect that their websites would provide up-to-date information including published outputs as well as some indication in relation to mechanisms of data access and sustainability. Adoption of the suggestion of Galsworthy et al to develop a central EU repository to make outputs permanently accessible for open meta-analysis and data reuse could improve this shortcoming.17 18 For initiatives whose data contribute to postauthorisation studies, their registration in the publicly accessible EU PAS Register provides an opportunity to increase the dissemination of methods and results.19
Proposed way forward
From our review, there is an apparent lack of an overarching vision for an infrastructure that would provide sustainable access to the data generated by the projects. Having said that, most recent initiatives include roadmaps as part of their deliverables which may potentially assist in delivering sustainability in the future. Moreover, the 11th Call for Proposals launched by the IMI 2 Joint Undertaking20 aims to provide solutions to ensure that significant results from IMI projects become fully exploitable, available to all relevant end users and/or fully sustainable in the long term and in their own right.
Based on our analysis, we highlight the following additional options that stakeholders including the European Commission, regulators, academia, sponsors and advocacy groups may wish to consider for ensuring better exploitation of funded initiatives:
Regulators and relevant end-users of the final deliverables should be involved early on in the planning and definition of the EU-funded initiatives.
To streamline efforts and resources and promote interoperability between outputs, consideration should be given to existing data and lessons learnt from past projects. This should help the identification of the real public health needs to be addressed by new initiatives.
If timescales allow, future EU RWE initiatives could take account of the EMA business pipeline on medicinal products to avoid gaps in the generation of RWD and ensure regulatory needs are supported.
Consideration should be given to the development of an agreed set of common data elements across data sources to promote harmonisation and interoperability, especially across those focused on the same therapeutic areas.
Use of real world data to support regulatory decision-making requires an in-depth understanding of the capture, characteristics, quality and validity of the data. Better quantification and information on these areas would enhance utility in the regulatory setting, for example, through the use of the available EMA procedure on qualification of novel methodologies for medicine development.21
The maintenance of the initiatives’ websites at the end of the funding period is key to ensure stakeholders are able to keep up to date on the progress made and on the deliverables/achievements. Maintenance of these websites should permit easy public access to information, related reports and peer-reviewed publications but would require dedicated funding.
There is a need for clarity on the possibilities to access RWD sources generated by the initiatives to allow their reuse in other contexts.
Consideration of explicit mechanisms to ensure the sustainability of outputs delivered by initiatives should be a priority and should be a requirement for any project proposal. The availability of funding options to support the ongoing maintenance of RWD resources would incentivise their development and ensure as wide an access as possible.
Conclusion
This inventory provides a better understanding of the extent of existing RWD resources emerging from EU-funded initiatives and highlights multiple shortcomings. Ideally, results of such initiatives should be reused and sustained and any lessons learnt disseminated. However, despite the potential of some initiatives to provide RWE that would support decision-making on the safety of medicinal products, there are challenges for their utilisation in a regulatory context due to the obstacles in the exploitation of their outputs (eg, limited data access, lack of sustainability).
Gaps and opportunities were identified in terms of specific therapeutic areas which may require RWD but were not a focus of funded initiatives. A number of solutions are proposed to enable better streamlining, communication and sustainability of the outputs generated through the EU-funded initiatives. As the 65 initiatives together were granted more than 734 million Euros of public funding, it is imperative that the shortcomings highlighted here are addressed in future funding programmes. This would go some way to ensuring delivery of stated objectives, data availability, sustainability and reflection of areas of medical need.
Acknowledgments
The authors thank Enrico Tognana at the European Medicines Agency for providing data on the EMA pharmaceutical business pipeline covering the period from March 2017 to December 2019, as well as the reviewers for the revision of this manuscript.
Footnotes
Contributors AC and XK conceived the idea of the study. All authors were involved in the study design. KP, PM and AP performed the analysis and interpretation of results. All authors contributed to writing the manuscript and approved the final draft.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Disclaimer The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the European Medicines Agency or one of its committees or working parties.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All the information provided in this manuscript other than the data on the EMA business pipeline is publically available online.