You are here

  • Home
  • Archive
  • Volume 7, Issue 8
  • Development and validation of hospital information system-generated indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults: the PACHA study protocol

Article Text

Article menu

Download PDFPDF

XML
Health services research
Protocol
Development and validation of hospital information system-generated indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults: the PACHA study protocol
  1. Aurélie Petit-Monéger1,2,
  2. Frantz Thiessard1,2,
  3. Vianney Jouhet1,2,
  4. Pernelle Noize2,3,
  5. Driss Berdaï3,
  6. Marion Kret1,
  7. Rémi Sitta1,
  8. Louis-Rachid Salmi1,2,
  9. Florence Saillour-Glénisson1,2
  10. The PACHA research group
  1. 1 CHU de Bordeaux, Pôle de santé publique, Service d’Information Médicale, Bordeaux, France
  2. 2 University of Bordeaux, ISPED, Centre INSERM U1219-Bordeaux Population Health, Bordeaux, France
  3. 3 CHU de Bordeaux, Pôle de santé publique, Service de Pharmacologie Médicale, Bordeaux, France
  1. Correspondence to Dr. Aurélie Petit-Monéger; aurelie.petit-moneger{at}u-bordeaux.fr

Abstract

Introduction The appropriateness of oral anticoagulant prescriptions is a major challenge to improve quality and safety of care. As indicators of the appropriateness of oral anticoagulant prescriptions are lacking, the aim of the study is to develop and validate a panel of such indicators, in hospitalised adults, from the hospital information system of two university hospitals in France.

Methods and analysis The study will be carried out in four steps: (1) a literature review to identify indicators of the appropriateness of oral anticoagulant prescriptions and their conditions of appropriateness; (2) a Delphi consensus method to assess the potential utility and operational implementation of the selected indicators; (3) techniques of medical data search to implement indicators from the hospital information system and; (4) a cross-sectional study to assess the ability of indicators to detect inappropriate oral anticoagulant prescriptions, performance of medical data search techniques for tracking or retrieving information and the ability of tools to be transferred into other institutions. The fourth step will include up to 80 patient hospital stays for each indicator, depending on the prevalence of inappropriate prescriptions estimated in interim analyses.

Ethics and dissemination This work addresses the current lack of quality indicators of the appropriateness of oral anticoagulant prescriptions. We aim to develop and validate such indicators for integrating them into hospital clinical practice, as part of a structured approach to improve quality and safety of care. As each hospital information system is different, we will propose tools transferable to other healthcare institutions to allow an automated construction of these indicators. The PACHA study protocol was approved by institutional review boards and ethics committees (CPP Sud-Ouest et Outre Mer III—DC 2016/119; CPP Ile-de-France II—CDW_2016_0014).

Registration details Clinical Trial.gov registration: NCT02898090.

  • Quality In Health Care
  • Public Health
  • Cardiology
  • Vascular Medicine

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

http://dx.doi.org/10.1136/bmjopen-2017-016488

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of the study

    • The study will address the current lack of indicators of the appropriateness of oral anticoagulant prescriptions, based on a comprehensive literature review and consensus of European experts.

    • The study will present a comprehensive approach to develop and validate indicators of the appropriateness of oral anticoagulants which can be used as a model in the future for the development and validation of quality and safety indicators automated from hospital information systems.

    • The study will propose transferable tools for the automatisation of quality indicators from the hospital information system of any healthcare institution.

    • The study will perform a semi-automated retrospective retrieval of eligible hospital stays for sampling design due to operational difficulty in implementing a large manually prospective retrieval within care units.

    • The study will be performed in an exploratory approach and will be limited to the assessment of about 15 indicators due to the time-consuming manually extraction of data within patient records for the reference test.

    Introduction

    Appropriateness of care, defined as the adequacy of any care to patient needs in accordance with practice guidelines, is a key dimension of quality of care and a major challenge for patient safety and efficiency of healthcare systems.1 Inappropriate care includes misuse, underuse and overuse.2–4 Chassin et al  reported that underuse frequency varies between 40% and 60% of all provided care,2 while Morgan et al reported that overuse frequency varies between 10% and 30%.3 Improving appropriateness of care is a priority in North America5 and Europe, including France,6 especially for its potential impact on quality and cost of care.

    Drug use holds a special place within appropriateness issues. The impact of drug adverse events is important; in France, they could be implicated in 20% of emergency room visits and in 3%–5% of hospitalisations,7–9 while 25% of patients admitted to intensive care units would suffer from at least one organ failure related to drug adverse event.10 Assessment of the appropriateness of drug prescriptions must ensure that patients receive the appropriate drug, for the appropriate indication, at an appropriate dosage, with an appropriate treatment adaptation depending on clinical and biological monitoring. Up to 30%–50% of serious adverse events related to drugs could be avoided11 12 and prevalence of drug adverse events concerns 10%–20% of hospitalised patients; more than 1 in 10 patients would present such an event during hospitalisation.13–15 The French national study of adverse events related to drugs (ENEIS 2) estimated the incidence of severe drug adverse events to 0.7 per thousand days of hospitalisation in 2009.9 Furthermore, international studies reported that 20%–60% of drug prescriptions could be inappropriate.16–18 Thus, the frequency, severity and impact of adverse events could be reduced by more appropriate prescriptions both in outpatient and hospitalised patients.

    Oral anticoagulants: a priority drug class

    Oral anticoagulants (vitamin K antagonists (VKA) and direct oral anticoagulants (DOA)) are among the drugs most frequently involved in adverse events.19 Epidemiological studies report that bleeding events related to VKA are among the first drug adverse events.7 11 12 In France, VKA could be involved in 17 000 hospitalisations per year, of which more than half could be avoided.20 VKA are especially difficult to use due to a narrow therapeutic index and the need for regular laboratory monitoring based on the International Normalised Ratio.20

    The development of indicators of oral anticoagulation prescriptions is especially justified by the recent arrival of DOAs, which share the same bleeding risk than VKA. Furthermore, the appropriateness of oral anticoagulant use and its economic impact is a major public health issue due to a large targeted population and frequent use.21

    Identified inappropriate practices of oral anticoagulant prescriptions at hospital may explain some bleeding or thrombotic complications9 22–24; such adverse events could be reduced by improving prescriptions and monitoring practices.25 Thus, the large targeted population, the high frequency of prescriptions and the strong risk of adverse events justify the importance of developing tools for improving the appropriateness of oral anticoagulant prescriptions in hospitalised patients.

    Importance of validated indicators of the appropriateness of oral anticoagulants

    Despite clear guidelines for clinical practice of oral anticoagulant prescriptions,20 21 26–32 validated indicators measuring the appropriateness of oral anticoagulant prescriptions are lacking. Such indicators could provide a way to regularly monitor oral anticoagulant prescriptions for validated clinical indications (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease) or clinical situations in patients under oral anticoagulants (trauma or planned surgery). Published studies mainly focused on the development of indicators on atrial fibrillation,33 especially on the indication of warfarin therapy, but did not assess their accuracy.34–39 The availability of indicators able to detect inappropriate oral anticoagulant prescriptions would improve the effectiveness and safety of prescriptions.

    Feedback of indicators of the appropriateness of oral anticoagulant prescriptions to health professionals could strengthen tools for self-assessment and quality improvement at hospital. As there is a need to provide regular feedback of indicators to health professionals to improve practices,40 it implies that such indicators may be automated from the hospital information system.

    Challenges

    The challenge is to be able to provide useful, implementable and valid final tools in any healthcare institution in France or elsewhere; this supposes the ability of tools to be transferred for developing indicators from other hospital information systems.

    Definition, selection and validation of indicators

    Because many guidelines refer to oral anticoagulant prescriptions for a wide range of clinical situations,41–45 there might be a large panel of indicators potentially measuring the appropriateness of oral anticoagulants. The construction of a dashboard of indicators requires defining these indicators, selecting and prioritising them according to their utility and interest for operational implementation, implementing them from the hospital information system and assessing their ability to detect inappropriate oral anticoagulant prescriptions. This assessment is justified by potential simplifications of the definition of indicators that might occur during their implementation, especially due to availability of the required data.

    Construction of indicators from a data warehouse and ability of tools to be transferred

    Information needed to establish a dashboard of indicators measuring the appropriateness of oral anticoagulant prescriptions is generally dispersed in many commercial applications (eg, biology results, imagery results, drug prescriptions or medical diagnoses coded with the International Classification of Diseases) within the hospital information system. These applications are not always interfaced, and the information system structure might differ between healthcare institutions. Moreover, data collected are often heterogeneous (structured or not, coded with international, national or only local terminologies) and redundant. Thus, there is a need to calculate the indicators from a data warehouse, which is a database used to collect, arrange and store information previously collected within the hospital information system through different commercial applications.

    We hypothesise that the integration of data from the hospital information systems of two French university hospitals into an i2b2 (Informatics for Integrating Biology and the Bedside) data warehouse will allow homogenising and structuring information contained in patient records. This should guarantee better performance of indicators to detect inappropriate oral anticoagulant prescriptions. The i2b2 is an open source data warehouse including a set of modules developed by Harvard Medical School to provide independent investigators with tools necessary to collect and manage project-related medical research data.46 As this interoperable software framework is used worldwide by a large scientific community and many hospitals, the implementation of this data warehouse will allow secondary use of data for all indicators, including those of other studies, without having to carry again the integration work of each of data sources.

    Aims of the PACHA study

    The main objective of the PACHA study (indicateurs de ‘Pertinence des prescriptions d’AntiCoagulants oraux à l’Hôpital Automatisés’) is to develop and validate indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults from the hospital information systems of two French university hospitals (Bordeaux University and Georges Pompidou European hospitals). As such, we aim to define a panel of useful, implementable, valid, reliable and robust indicators to alert on the existence of inappropriate prescriptions. These indicators would cover the main clinical situations of oral anticoagulant prescriptions and could be used by any health professional interested in strengthening tools for self-assessment and quality improvement at hospital.

    Methods and analysis

    The PACHA study protocol was approved by institutional review boards and ethics committees (CPP Sud-Ouest et Outre Mer III—DC 2016/119; CPP Ile-de-France II—CDW_2016_0014) and registered in Clinical Trial (registration number: NCT02898090). The steering and scientific committees of PACHA include epidemiologists, hospital information system specialists, pharmacologists, pharmacists, cardiologists, neurologists, geriatricians, emergency physicians, anaesthesiologists, specialists in biology or haematology, and statisticians.

    Structure of the study

    The study will be carried out in four steps (Figure 1): (1) identification of possible indicators measuring the appropriateness of oral anticoagulant prescriptions and their conditions of appropriateness; (2) selection by experts of a panel of indicators judged both useful and implementable; (3) construction of indicators from the hospital information system; (4) assessment of the ability of selected indicators to detect inappropriate oral anticoagulant prescriptions, of the performance of medical data search techniques for tracking or retrieving information needed for the construction of indicators, and of the ability of tools to be transferred to other institutions. The four steps will be carried out, in parallel, at Bordeaux University Hospital (Bordeaux, France) and Georges Pompidou European Hospital (Paris, France), while Rennes University Hospital (Rennes, France) will only be involved in the fourth step of the study (table 1). These three investigating centres are university hospitals combining multidisciplinary hospital, academic and research activities; in 2015, tens of thousands of patients were hospitalised in each hospital.

    Figure 1
    Figure 1

    Flow chart of the PACHA study.

    View this table:
    Table 1

    Description of the main methodological characteristics of the PACHA study in four steps

    Step 1: Identification of indicators of the appropriateness of oral anticoagulant prescriptions

    Study design

    A systematic literature review will be performed to identify indicators of the appropriateness of oral anticoagulant prescriptions in hospitalised adults and their conditions of appropriateness. These indicators will thus be designed to identify inappropriate prescriptions of oral anticoagulants in hospitalised patients.

    Literature search

    The literature review will identify guidelines, from European high authorities for health and European learned societies, focusing on clinical situations for which a prescription of oral anticoagulants (VKA or DOAs) is indicated or contraindicated, and focusing on the appropriate use of oral anticoagulants for validated clinical indications in terms of dosage, duration of treatment, pretherapeutic assessment, compliance with contraindications and monitoring during treatment. The targeted clinical situations will be: (i) frequent cardiovascular situations with thromboembolic risk that could justify a prescription of oral anticoagulants: atrial fibrillation (estimated prevalence between 1.5% and 2.0% of the general population),41 valvular heart disease and prosthetic heart valve (prevalence of valvular heart disease estimated at 2.5% of the general population, of whom a quarter of them benefit from a prosthesis)45 47 48 and venous thromboembolic disease (estimated annual incidence between 100 and 200 per 100 000 persons in the general population),43 for which misuse, underuse or overuse of oral anticoagulant as well as inappropriate implementation of guidelines might occur22–24 49–52; (ii) trauma or planned surgery in patients under oral anticoagulants42 53 for which an increased haemorrhagic risk due to inappropriate adaptations of oral anticoagulants might occur.

    Measures and analysis

    We will perform a critical appraisal of all selected guidelines. We then will define a panel of indicators of the appropriateness of oral anticoagulants. For each indicator, we will specify its name, objective, construction (numerator and denominator for proportions), domain of application (inclusion and exclusion criteria) and methods for data collection (data sources and sampling). Appropriateness of care will be defined for each indicator in terms of drug dosage, duration of treatment, pretherapeutic assessment, compliance with contraindications and monitoring during treatment.

    Expected results

    The identified indicators should cover misuse, underuse and overuse of oral anticoagulants. Indicators of misuse will refer to patients presenting a clinical situation for which an oral anticoagulant is indicated and has been prescribed, but for which prescribed modalities or choice of the anticoagulant do not respect: (1) strict contraindications or contraindicated associations; (2) dosage, rhythm or duration of oral anticoagulant treatment; (3) conditions for pretherapeutic assessment and monitoring of oral anticoagulant; (4) adaptations required during treatment, especially in case of trauma or planned surgery requiring to stop temporarily or permanently the oral anticoagulant, a change in dosage, rhythm or duration of prescription, or a relay towards injectable anticoagulants. Indicators of underuse will refer to patients with a clinical situation for which no oral anticoagulant has been prescribed, despite an indication for such a prescription and the absence of contraindication for oral anticoagulants. Indicators of overuse will refer to patients who have been prescribed an oral anticoagulant despite the absence of a clinical situation justifying such a prescription, or when the benefits of the treatment do not exceed the risks.

    Step 2: Selection of useful and implementable indicators

    Study design

    In accordance with guidelines,54 a Delphi consensus method54–56 in two rounds will be conducted among experts to assess the potential utility and operational implementation of the indicators among those previously identified during the first step of the study.

    Expert selection

    We will select experts with clinical expertise about the targeted clinical situations and prescriptions of oral anticoagulants, while not being members of the study steering nor scientific committees. These experts will be specialists in cardiology, vascular medicine, neurology, geriatrics, emergency medicine, anaesthesia and intensive care, pharmacology or haematology, practising in France, Belgium or Switzerland. Experts meeting these criteria will be selected by members of the study scientific committee from their professional networks and lists of authors who have participated in the development of European guidelines focusing on prevention or treatment of the targeted clinical situations or focusing on appropriate use of oral anticoagulants for validated clinical indications.

    Measures and analysis

    Experts will be asked to judge the potential utility (defined as the indicator ability to detect inappropriate oral anticoagulant prescriptions and to implement improvement actions) and operational implementation in terms of frequency and severity (defined as the indicator ability to detect inappropriate oral anticoagulant prescriptions that are frequent or severe enough to justify changing practices). Experts will be asked to rate utility and operational implementation of the proposed indicators on a qualitative binary scale (yes or no). Experts will also be asked to give their opinion on the way the proposed indicators are defined and formulated, especially for some definitions or criteria that are not consensual in the literature. Each expert will also have the opportunity to propose up to three new clearly defined indicators, and to specify from which guidelines they have been identified.

    At each round, an indicator will be selected if at least 8 out of 10 experts considered the indicator both useful and implementable (strong consensus for selection). An indicator will be deleted if at least 8 out of 10 experts considered the indicator both not useful and not implementable (strong consensus for deletion).  All expert ratings will be reported in an anonymised database.

    Expected results

    At the end of the two rounds of the Delphi process, a final synthesis meeting will be organised by the steering committee to validate the final list of selected indicators, especially by considering the indicators for which no consensus will have been obtained. We will also establish the order in which the final panel of indicators will be implemented from the hospital information system and assessed for their ability to detect inappropriate oral anticoagulant prescriptions.

    Step 3: Construction of the selected indicators from the hospital information system

    Study design

    During this prospective phase, techniques and tools of medical data search will be applied to implement the selected indicators from the hospital information system.

    Study population

    The study population will include hospital stays of patients aged 18 years and over, cared at Bordeaux University or Georges Pompidou European hospitals for targeted clinical situations (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease, trauma or planned surgery in patients under oral anticoagulants) or having had a prescription of oral anticoagulants during their hospital stay in medicine, surgery, emergency or postemergency unit, from 1 January 2015 to 31 December 2015, with medico-administrative data available in the hospital information system (electronic patient record). Hospital stays of patients who would have denied the computer processing of their electronic record will not be included in the study.

    Study samples

    For each indicator, the medical information units of Bordeaux University and Georges Pompidou European hospitals will carry out a semi-automated retrospective retrieval of eligible patient hospital stays. This retrospective retrieval will be carried out from the hospital information system: (i) from queries listing patient hospital stays concerned by the targeted clinical situations within electronic patient records (codes defined according to the International Classification of Diseases 10th version); (ii) from queries listing patient hospital stays during which a VKA or DOA has been prescribed within the drug prescription commercial application (codes defined according to the Anatomical, Therapeutic and Chemical Classification of Drugs, ATC) or (iii) from these two types of queries.

    A first series of samples of patient hospital stays (type 1 samples) will be constituted for constructing each of the selected indicators (one type 1 sample per indicator). These type 1 samples might be stratified, if deemed necessary, on up to three out of five possible stratification variables (type of hospital stay; type of prescribed oral anticoagulant; age and sex of patients; type of trauma) depending on the indicator. Indicators will be considered independently of each other.

    Three additional samples of patient hospital stays (samples of types 2, 3 and 4) will be constituted to investigate hospital stays which could have been missed by queries within the hospital information system during the retrospective retrieval (one sample of types 2, 3 and 4 per indicator). These samples might also be similarly stratified, if deemed necessary, up to three stratification variables.

    For indicators whose lists of patient hospital stays will be identified directly from queries, we will proceed in sampling of the required number of patient hospital stays in each investigating centre. For indicators whose sampling will require investigation of the randomly selected patient hospital stays, clinical research technicians will investigate patient files to ensure eligibility; eligible stays will be included in the corresponding sample according to the above-mentioned procedure. The procedure will be performed following the order of prioritisation for the construction of indicators (as defined at the end of the study step 2) and will be continued until the required number of patient hospital stays is obtained for each sample of each indicator; a real-time monitoring of inclusions will be performed.

    Measures and analysis: construction and calculation of indicators from the data warehouse

    Information needed for the construction of the selected indicators from the hospital information system will be integrated into the specific i2b2 data warehouse46 in both the Bordeaux University and Georges Pompidou European hospitals. The indicators of the appropriateness of oral anticoagulants prescriptions will be constructed from this data warehouse, which will allow answering quickly to queries that may involve millions of patient records, while constructing indicators in other healthcare institutions using the same data warehouse. The data warehouse will both contain structured data (eg, medical diagnoses, acts, drug prescriptions, biological prescriptions and their results, prescriptions of radiology) and unstructured data (data only available in a textual format).

    Concepts of interest that will need to be available in the data warehouse for the construction and calculation of each indicator will be defined during the first and second steps of the study. We will need to identify, at least, clinical data, symptoms or diagnoses, drugs, laboratory tests, imaging tests or procedures for indirectly identifying a diagnosis or an adverse effect (Figure 2). Concepts of interest related to structured data will be automatically extracted from the data warehouse. Other techniques of medical data search will be used to identify concepts of interest related to unstructured data from the data warehouse, especially Natural Language Processing (NLP)57 58; NLP techniques will be needed for identifying concepts of interest in text areas (eg, targeted clinical situations, drug treatments, treatment monitoring, comorbidities or adverse events). To improve the ability of the NLP techniques to retrieve the concepts of interest, we will use a morpho-syntax analyser, which will allow identifying, in free-text boxes, the syntactic role of each word (namely verb, adjective or name). We will develop a version of the module of NLP that uses the morpho-syntax analyser and another that will directly process raw text. Analysis of each concept will be reinforced by a judgement of its contextual relevance, taking into account negation (whether it is a concept really present or described as absent) and temporality (whether the concept is present for a patient at the time t of investigation). Both Bordeaux University and Georges Pompidou European hospitals will extract data needed for the construction of indicators from an i2b2 data warehouse, especially by developing and using an NLP module that will be integrated into the tools associated with this data warehouse.

    Figure 2
    Figure 2

    Concepts of interest needed to be available in the data warehouse for the construction of indicators.

    All indicators will be calculated from patient hospital stays of type 1 samples. Patient hospital stays which would have been missed by queries during the retrospective retrieval within sample types 2, 3 and 4 will be only taken into account for calculating indicators in the context of a robustness analysis presented in the fourth step of the study. A limited number of priority indicators will be commonly constructed at both hospitals; half of the other indicators will be constructed in Bordeaux University Hospital and the other half will be constructed in Georges Pompidou European Hospital.

    Expected results

    The expected results will be values of indicators.

    Step 4:Ability of the indicators to detect inappropriate prescriptions, performance of medical data search techniques for tracking or retrieving information and ability of tools to be transferred to other institutions

    A) Assessment of the ability of indicators to detect inappropriate prescriptions

    Study design

    The study design will be a cross-sectional study.

    Study population and samples

    The study population will be the same as those previously described for the construction of the selected indicators (step 3 of the study). All samples previously described will be used (samples of types 1, 2, 3 and 4).

    Measures and analysis: data collection for analysis of accuracy, reliability and robustness of indicators

    Data for analysis of accuracy, reliability and robustness of indicators will be collected at both hospitals for common indicators. The measure of accuracy, reliability and robustness will be conducted according to methodological standards.56 59–62

    Data collection for the accuracy analysis

    Data collection and analysis of accuracy will be carried out independently for each indicator. For each patient of a given type 1 sample, the value of the indicator will be calculated: (i) from data related to patient hospital stays that will be manually extracted by the clinical research technicians (one for each investigating centre) within electronic patient records; (ii) from data related to patient hospital stays that will be automatically extracted from the data warehouse. Information manually extracted by the clinical research technicians will be collected on a specific electronic case report form. The prevalence of inappropriate oral anticoagulant prescriptions could thus be estimated in real-time from data collected by the clinical research technicians as well as from data automated from the data warehouse.

    The reference defining the actual appropriateness of prescriptions will be defined from information manually extracted by the clinical research technicians in patient records, and then, secondly, by the judgement of clinical experts based on information extracted by the clinical research technicians, patient records and clinical expertise. In each investigating centre, a same group of three experts will be responsible for judging the appropriateness of prescriptions for each patient hospital stay of the sample constituted for a given indicator. In case of disagreement between the three experts, the opinion of the majority of them will be considered.

    Data collection for the reliability analysis

    For each indicator, patient records will be randomly drawn from subpopulations concerned by sources of coding variations. The results of the appropriateness measure will then be compared between two different time periods of coding or between two units with different coding practices.

    Data collection for the robustness analysis

    For each indicator, patient records will be randomly selected and results of the appropriateness measure will be compared by varying their conditions of appropriateness (as defined at the end of the first step) or by using or not techniques of NLP (without using NLP; with tools of NLP integrating the morpho-syntax analyser and; with tools of NLP directly processing raw text).

    Similarly, for each indicator, results of the appropriateness measure will be compared taking into account or not patient hospital stays that would have been missed by the hospital information system among the samples of types 2, 3 and 4 investigated by the clinical research technicians. For each patient hospital stay of these samples, the indicator value will be produced by the automated system. In parallel, the clinical research technicians will carry out a manual extraction of information needed to measure the appropriateness, by a complete review of the patient hospital stays; this information will be collected on a specific electronic case report form.

    Statistical parameters

    The primary statistical parameter will focus on the prevalence of inappropriate prescriptions of oral anticoagulants estimated, for each indicator, from data warehouse in comparison to the prevalence estimated from the reference (information manually extracted by the clinical research technicians in the patient hospital stay), and the estimation of its 95% CI.

    The secondary statistical parameters will focus on:

    • Accuracy of each indicator (sensitivity, specificity and predictive values) to detect inappropriate oral anticoagulant prescriptions from data related to patient hospital stays that will be automatically extracted from the data warehouse compared with the reference.

    • Reliability of each indicator (Kappa coefficients for unordered qualitative indicators, weighted Kappa coefficients for ordered qualitative indicators and intraclass coefficients for quantitative indicators) by analysing their results’ variations depending on variations of information coding in the hospital information system.

    • Robustness of each indicator (Kappa coefficients for unordered qualitative indicators, weighted Kappa coefficients for ordered qualitative indicators and intraclass coefficients for quantitative indicators) by analysing their results’ variations depending on variations of their conditions of appropriateness, or the use or not of techniques of NLP, or by taking into account or not patient hospital stays that would have been missed within the hospital information system among the samples of types 2, 3 and 4 investigated by the clinical research technicians.

    Sample size

    For the accuracy analysis, the required sample size of 80 patient hospital stays has been calculated to estimate a prevalence of 5%, with a 95% CI of 1.4% to 12.3% according to the exact binomial distribution. In the perspective of warning indicators, we considered a prevalence of 5% to guarantee a minimum precision for low prevalence of inappropriate prescriptions that are severe enough to justify changing practices. Furthermore, by considering an exploratory approach, an interim analysis will be carried out at the end of investigation of the first 40 patient hospital stays to optimise the balance between the expected precision and required resources for data collection. If the observed number of patient hospital stays with inappropriate prescription is ≥10 (corresponding to prevalence ≥25%), data collection will be stopped for this indicator since the increase in the number of patient hospital stays would not significantly increase precision of estimates in comparison to required resources. On the contrary, if the observed number of patient hospital stays with inappropriate prescription is <10 (corresponding to prevalence strictly <25%), data collection will be continued until 80 patient hospital stays, as initially planned. This sample size of 40 patient hospital stays has been fixed according to the 95% CI of the exact binomial distribution, which is 12.7% to 41.2% with a prevalence of 25%.

    For the robustness analysis, the estimation of the number of patient hospital stays required for samples of types 2, 3 and 4 of each indicator is based on statistical hypotheses derived from the lot quality assurance sampling technique63 64 (null hypothesis (H0): the group of stays is not acceptable if the proportion of patient hospital stays that have been missed by the hospital information system in the sample is greater than or equal to the defined value P0; alternative hypothesis (H1): the group of stays is acceptable if the proportion of patient hospital stays that have been missed by the hospital information system is strictly lower than the defined value P0; N: size of the lot, which corresponds to the population resulting from the medical diagnoses queries for additional patient hospital stays needing to be investigated to detect any missed patient hospital stays; d+1= number of missed patient hospital stays whose observation on the sample would imply to reject the group of stays). Using tables for lot quality assurance sampling technique at 1 df,63 the number of patient hospital stays required is 25 with the following hypotheses: P0=10%, α=5%, d=0 and size of the lot between 100 and 1 00 000 patient hospital stays.

    Expected results

    The scientific committee of the study will proceed to a collective validation of indicators, by taking into account their ability to detect inappropriate prescriptions in relation to criteria of validation that will have been defined a priori.

    B) Assessment of performance of medical data search techniques for tracking or retrieving information

    Study design

    The study design will be a cross-sectional study.

    Study population and samples

    The study population will be the same as those previously described for the construction of the selected indicators (step 3 of the study).

    The study samples will focus on the type 1 samples previously described, as well as on a supplementary sample of 50 randomly selected patient hospital stays (type 5 sample). The latter will be used, for each indicator, to assess the ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions covered by indicators in comparison to retrieval conducted without the data warehouse from electronic patient records and the drug prescription commercial application.

    For the constitution of the type 5 sample, we will identify all patient records identified in the data warehouse for which the concerned clinical situation or prescription of oral anticoagulant is present, on the same period than those during which the type 1 samples will have been selected. Within this group of patient records, we will remove those that will have already been included in type 1 samples; we will then randomly select 50 patient records from these subgroups.

    Measures and analysis

    For the assessment of performance of tools tracking information for the construction of indicators from patient hospital stays included in the type 1 samples, we will estimate three statistical parameters during primary analysis: recall (defined as the number of concepts that are correctly returned by the evaluated tool in comparison to the number of concepts returned by the reference), precision (defined as the number of concepts that are correctly returned by the evaluated tool in comparison to the total number of concepts returned by the search engine for a given query) and F-measure (which combines recall and precision).57 This assessment will be performed by comparing information extracted from the data warehouse with information manually extracted by the clinical research technicians from patient hospital stays. It will especially be carried out for the concepts that will have been identified by the NLP module. An analysis of recall, precision and F-measure will be performed according to the level of NLP used during search of concepts: (i) no use of NLP module; (ii) use of NLP module integrating the morpho-syntax analyser; (iii) use of NLP module directly processing the raw text. A secondary qualitative analysis will be performed by another independent clinical research technician who will not have participated to manual extraction of data. It will aim at assessing discordances between information extracted from the data warehouse and information manually extracted by the clinical research technicians in patient records for each indicator.

    To assess the ability of the data warehouse to identify targeted clinical situations or oral anticoagulant prescriptions, covered by indicators, from patient hospital stays included in the type 5 samples, we will perform another secondary qualitative analysis. This assessment will aim at analysing discordances between the retrieval of targeted clinical situations and/or oral anticoagulant prescriptions by two compared methods: (1) without using the i2b2 data warehouse, only on the basis of coded medical diagnoses and oral anticoagulant prescriptions from commercial applications and (2) by using the i2b2 data warehouse.

    Expected results

    The scientific committee of the study will be provided with useful information on performance of medical data search techniques using or not a data warehouse and NLP module.

    C) Assessment of the ability of tools to be transferred to other institutions

    Study design

    The study design will be a cross-sectional study.

    Study sample

    The study sample will focus on the hospital stays of patients aged 18 years and over, cared at Rennes University Hospital, between 1 January 2015 and 31 December 2015, for targeted clinical situations (atrial fibrillation, valvular heart disease and prosthetic heart valve, venous thromboembolic disease, trauma or planned surgery in patients under oral anticoagulants) or having had a prescription of oral anticoagulants during their hospital stay in medicine, surgery, emergency or postemergency unit, whose medico-administrative data will have been included in the eHOP data warehouse. Hospital stays of patients who would have denied the computer processing of their electronic record will not be included in this analysis.

    Measures and analysis

    Rennes University Hospital will extract the structured and unstructured data needed for the construction of these indicators from their eHOP data warehouse.65 Unstructured data will be extracted by using the NLP module that would have been especially implemented in a local web-service for improving the ability of tools to be transferred, with or without a morpho-syntax analyser. For this assessment, the primary statistical parameter will focus on the proportion of indicators that will be successfully automated.

    Expected results

    The validation of the ability of tools to be transferred will focus on an expected proportion of indicators successfully automated that will have been defined a priori.

    Ethics and dissemination

    This study will improve the quality and safety of care by addressing the current lack of indicators measuring the appropriateness of oral anticoagulant prescriptions, in France and Europe, based on guidelines for clinical practices in validated clinical indications. Such improvements will exceed the scope of healthcare institutions, since a high number of oral anticoagulant prescriptions initiated in such institutions are pursued in outpatients.

    We will propose a panel of useful, implementable, valid, reliable and robust indicators of the appropriateness of oral anticoagulant prescriptions that will be automated from the hospital information system and generalisable to other healthcare institutions in France and Europe. These indicators will be gathered in dashboards whose impact will then be assessed based on clinical and medico-economic criteria, which will improve the ability of healthcare institutions answering to current institutional requirements in quality of care.

    Validated indicators will be regularly conveyed to prescribing health professionals by individual, electronic, visual graphic feedbacks,66 as part of a continuous process of professional practices improvement. Such feedbacks will help the familiarisation of health professionals with quality indicators by integrating them into the heart of a dialogue on practices. Thus, they will be important tools for updating their knowledge about the appropriateness of oral anticoagulant prescriptions.

    In the future, the validated indicators would potentially be assessed for their sensitivity to change56 through the implementation of interventions to improve the appropriateness of oral anticoagulant prescriptions. We can also consider collaborative work to generalise these indicators and develop continuity of care between healthcare institutions and ambulatory medicine.

    At last, this study will propose transferable tools to other French or European healthcare institutions to allow an automatisation of indicators of the appropriateness of oral anticoagulants as well as future other quality and safety indicators.

    Conclusion

    This study addresses the current lack of indicators of the appropriateness of oral anticoagulant prescriptions, based on guidelines and validated clinical indications. We aim at integrating them into hospital clinical practice, as part of a structured approach to improve quality and safety of care.

    By selecting useful and implementable indicators from judgement of European experts and by providing transferable tools for their automatisation from the hospital information system of other healthcare institutions, we will strengthen their potential of generalisation in French and European healthcare institutions to improve quality and safety of care. Such ability of tools to be transferred between different contexts and countries is important to improve the effectiveness of quality of care strategies based on the use of indicators.67 Furthermore, this study will provide a comprehensive model for the development and validation of other indicators of the appropriateness of care which may be automated from hospital information systems.

    Acknowledgments

    The authors would like to thank the experts who have agreed to participate to the Delphi consensus: Cardiology: Heidbuchel Hein (University of Leuven, Belgium), Iung Bernard (Hopital Bichat, France), Meneveau Nicolas (University Hospital of Besancon, France); Vascular medicine: Pernod Gilles (University Hospital of Grenoble, France), Reny Jean-Luc (University Hospital of Geneva, Switzerland); Neurology: Cordonnier Charlotte (University Hospital of Lille, France), Calvet David (Centre Hospitalier Saint-Anne, France); Geriatrics: Toulza Olivier (University Hospital of Toulouse, France), Friocourt Patrick (Centre Hospitalier de Blois, France), Belmin Joël (Hopital Charles-Foix, France); Emergency medicine: Carpentier Françoise (University Hospital of Grenoble, France), Jehle Eric (University Hospital of Toulouse, France), Honnart Didier (University Hospital of Dijon, France); Anaesthesiology: Rosencher Nadia (Hopital Cochin, France), Godier Anne (Fondation Rothschild, France), Steib Annick (University Hospital of Strasbourg, France), Longrois Dan (Hopital Bichat, France), Faraoni David (Boston Children’s Hospital, USA); Pharmacology: Montastruc Jean-Louis (University Hospital of Toulouse, France), Laine-Cessac Pascale (University Hospital of Angers, France); Haematology: Gouin-Thibault Isabelle (Hopital Charles-Foix, France), Fontana Pierre (University Hospital of Geneva, Switzerland).

    References

    1. 1.↵
       Haute Autorité de Santé. Evaluer La pertinence des soins : une démarche de progrès [Internet]. Saint-Denis La Plaine: Haute Autorité de Santé, 2012.Available at:http://www.has-sante.fr/portail/jcms/c-1197428/fr/evaluer-la-pertinence-des-soins-une-demarche-de-progres (accessed 2 Jan 2017)
    2. 2.↵
      2. Chassin MR ,
      3. Galvin RW
      . The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality. JAMA. 1998;280:1000–5.doi:10.1001/jama.280.11.1000
      OpenUrlCrossRefPubMedWeb of Science
    3. 3.↵
      2. Morgan DJ ,
      3. Brownlee S ,
      4. Leppin AL , et al
      . Setting a research agenda for medical overuse. BMJ. 2015;351:h4534.doi:10.1136/bmj.h4534
      OpenUrlFREE Full Text
    4. 4.↵
       Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: The National Academies Press, 2001.
    5. 5.↵
      2. Berwick DM ,
      3. Hackbarth AD
      . Eliminating waste in US health care. JAMA. 2012;307 (14):1513–6.doi:10.1001/jama.2012.362
      OpenUrlCrossRefPubMedWeb of Science
    6. 6.↵
       Direction Générale de l’Offre de Soins. Guide méthodologique pour l’amélioration de la pertinence des soins. Paris: Ministère des Affaires sociales et de la Santé, 2012.
    7. 7.↵
      2. Pouyanne P ,
      3. Haramburu F ,
      4. Imbs JL , et al
      . Admissions to hospital caused by adverse drug reactions: cross sectional incidence study. French Pharmacovigilance Centres. BMJ. 2000;320 (7241):1036.
      OpenUrlFREE Full Text
    8. 8.↵
      2. Queneau P ,
      3. Bannwarth B ,
      4. Carpentier F , et al
      . Emergency department visits caused by adverse drug events: results of a french survey. Drug Saf. 2007;30 (1):81–8.
      OpenUrlCrossRefPubMedWeb of Science
    9. 9.↵
      2. Michel P ,
      3. Minodier C ,
      4. Lathelize M , et al
      . Les événements indésirables graves associés aux soins observés dans les établissements de santé. Dossiers Solidarité et Santé, n°17 2010;.
    10. 10.↵
      2. Grenouillet-Delacre M ,
      3. Verdoux H ,
      4. Moore N , et al
      . Life-threatening adverse drug reactions at admission to medical intensive care: a prospective study in a teaching hospital. Intensive Care Med. 2007;33 (12):2150–7.doi:10.1007/s00134-007-0787-8
      OpenUrlCrossRefPubMedWeb of Science
    11. 11.↵
      2. van der Hooft CS ,
      3. Dieleman JP ,
      4. Siemes C , et al
      . Adverse drug reaction-related hospitalisations: a population-based cohort study. Pharmacoepidemiol Drug Saf. 2008;17 (4):365–71.doi:10.1002/pds.1565
      OpenUrlCrossRefPubMed
    12. 12.↵
      2. Bénard-Laribière A ,
      3. Miremont-Salamé G ,
      4. Pérault-Pochat M-C , et al
      . Incidence of hospital admissions due to adverse drug reactions in France: the EMIR study. Fundam Clin Pharmacol. 2015;29 (1):106–11.
      OpenUrl
    13. 13.↵
      2. Cecile M ,
      3. Seux V ,
      4. Pauly V , et al
      . [Adverse drug events in hospitalized elderly patients in a geriatric medicine unit: study of prevalence and risk factors]. Rev Med Interne. 2009;30 (5):393–400.doi:10.1016/j.revmed.2009.01.010
      OpenUrlPubMed
    14. 14.↵
      2. Miguel A ,
      3. Azevedo LF ,
      4. Araújo M , et al
      . Frequency of adverse drug reactions in hospitalized patients: a systematic review and meta-analysis. Pharmacoepidemiol Drug Saf. 2012;21 (11):1139–54.doi:10.1002/pds.3309
      OpenUrlPubMed
    15. 15.↵
      2. Imbs JL ,
      3. Pouyanne P ,
      4. Haramburu F , et al
      . [Iatrogenic medication: estimation of its prevalence in french public hospitals. Regional Centers of Pharmacovigilance]. Thérapie. 1999;54 (1):21–7.
      OpenUrlPubMed
    16. 16.↵
      2. Paillé-Ricolleau C ,
      3. Lejus C ,
      4. Potel G , et al
      . [Appropriateness of prescriptions of intravenous paracetamol at Nantes Teaching Hospital: a follow-up indicator for quality improvement]. Presse Med. 2010;39 (4):511–2.doi:10.1016/j.lpm.2009.11.012
      OpenUrlPubMed
    17. 17.↵
      2. Napolitano F ,
      3. Izzo MT ,
      4. Di Giuseppe G , et al
      . Frequency of inappropriate medication prescription in hospitalized elderly patients in Italy. PLoS One. 2013;8:e82359.doi:10.1371/journal.pone.0082359
    18. 18.↵
      2. Wright JD ,
      3. Neugut AI ,
      4. Ananth CV , et al
      . Deviations from guideline-based therapy for febrile neutropenia in cancer patients and their effect on outcomes. JAMA Intern Med. 2013;173 :559–68.doi:10.1001/jamainternmed.2013.2921
      OpenUrlCrossRefPubMed
    19. 19.↵
       Haute Autorité de Santé. Comment améliorer la qualité et la sécurité des prescriptions de médicaments chez la personne âgée?. Saint-Denis La Plaine: Haute Autorité de Santé, 2014.
    20. 20.↵
       Agence Nationale de Sécurité du Médicament et des Produits de santé. Bon usage des médicaments antivitamine K (AVK). Saint-Denis: Agence Nationale de Sécurité du Médicament et des Produits de santé, 2012.
    21. 21.↵
       Agence Nationale de Sécurité du Médicament et des Produits de santé. Les anticoagulants en France en 2014 : état des lieux, synthèse et surveillance. Saint-Denis: Agence Nationale de Sécurité du Médicament et des Produits de santé, 2014.
    22. 22.↵
      2. Ferro D ,
      3. Loffredo L ,
      4. Polimeni L , et al
      . Underuse of oral anticoagulants in patients with nonvalvular atrial fibrillation in Italy. Intern Emerg Med. 2007;2 (1):24–8.doi:10.1007/s11739-007-0005-2
      OpenUrlPubMed
    23. 23.↵
      2. Armbruster AL ,
      3. Buehler KS ,
      4. Min SH , et al
      . Evaluation of dabigatran for appropriateness of use and bleeding events in a community hospital setting. Am Health Drug Benefits. 2014;7 (7):376–84.
      OpenUrl
    24. 24.↵
      2. Tellor KB ,
      3. Patel S ,
      4. Armbruster AL , et al
      . Evaluation of the appropriateness of dosing, indication and safety of rivaroxaban in a community hospital. J Clin Pharm Ther. 2015;40 (4):447–51.doi:10.1111/jcpt.12288
      OpenUrl
    25. 25.↵
       Caisse Nationale de l’Assurance Maladie des Travailleurs Salariés. Nouveaux anti-coagulants oraux : une étude de l'Assurance Maladie souligne la dynamique forte de ces nouveaux médicaments et la nécessité d'une vigilance accrue dans leur utilisation. Paris: Caisse Nationale de l’Assurance Maladie des Travailleurs Salariés, 2013.
    26. 26.↵
       Haute Autorité de Santé. Avis commission de la transparence PREVISCAN®. Saint-Denis La Plaine: Haute Autorité de Santé, 2011.
    27. 27.↵
       Haute Autorité de Santé. Avis commision de la transparence COUMADINE®. Saint-Denis La Plaine: Haute Autorité de Santé, 2012.
    28. 28.↵
       Haute Autorité de Santé. Avis commision de la transparence SINTROM® et MINI-SINTROM®. Saint-Denis La Plaine: Haute Autorité de Santé, 2010.
    29. 29.↵
       Haute Autorité de Santé. Point sur l’utilisation des nouveaux anticoagulants oraux [Internet]. Saint-Denis La Plaine: Haute Autorité de Santé, 2013.Available at:http://www.has-sante.fr/portail/jcms/c-1700943/fr/point-sur-lutilisation-des-nouveaux-anticoagulants-oraux (accessed 2 Jan 2017)
    30. 30.↵
       Haute Autorité de Santé. Avis commission de la transparence ELIQUIS®. Saint-Denis La Plaine: Haute Autorité de Santé,, 2014.
    31. 31.↵
       Haute Autorité de Santé. Avis commission de la transparence PRADAXA®. Saint-Denis La Plaine: Haute Autorité de Santé,, 2014.
    32. 32.↵
       Haute Autorité de Santé. Avis commission de la transparence XARELTO®. Saint-Denis La Plaine: Haute Autorité de Santé,, 2014.
    33. 33.↵
      2. Rose AJ ,
      3. Berlowitz DR ,
      4. Frayne SM , et al
      . Measuring quality of oral anticoagulation care: extending quality measurement to a new field. Jt Comm J Qual Patient Saf. 2009;35 (3):146–55.doi:10.1016/S1553-7250(09)35019-9
      OpenUrlPubMed
    34. 34.↵
      2. McGlynn EA ,
      3. Asch SM ,
      4. Adams J , et al
      . The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–45.doi:10.1056/NEJMsa022615
      OpenUrlCrossRefPubMedWeb of Science
    35. 35.↵
      2. McGlynn EA ,
      3. Asch SM ,
      4. Adams JL , et al
      ; Appendix: The Quality of Health Care Delivered to Adults in the United States, 2006.
    36. 36.↵
      2. Cheng EM ,
      3. Fung CH
      . Quality indicators for the care of stroke and atrial fibrillation in vulnerable elders. J Am Geriatr Soc. 2007;55((Suppl 2)):S431–7.doi:10.1111/j.1532-5415.2007.01352.x
      OpenUrlCrossRefPubMedWeb of Science
    37. 37.↵
      2. Shrank WH ,
      3. Polinski JM ,
      4. Avorn J
      . Quality indicators for medication use in vulnerable elders. J Am Geriatr Soc. 2007;55((Suppl 2)):S373–82.doi:10.1111/j.1532-5415.2007.01345.x
      OpenUrlCrossRefPubMed
    38. 38.↵
      2. Wenger NS ,
      3. Shekelle PG
      . Assessing care of vulnerable elders: ACOVE project overview. Ann Intern Med. 2001;135: (8 Pt 2) 642–6.doi:10.7326/0003-4819-135-8_Part_2-200110161-00002
      OpenUrl
    39. 39.↵
      2. Estes NA ,
      3. Halperin JL ,
      4. Calkins H , et al
      . ACC/AHA/Physician Consortium 2008 Clinical Performance measures for adults with Nonvalvular Atrial Fibrillation or Atrial Flutter: a report of the American College of Cardiology/American Heart Association Task Force on Performance measures and the physician Consortium for Performance Improvement (Writing Committee to Develop Clinical Performance measures for Atrial Fibrillation) Developed in collaboration with the Heart Rhythm Society. J Am Coll Cardiol. 2008;51 (8):865–84.doi:10.1016/j.jacc.2008.01.006
      OpenUrlFREE Full Text
    40. 40.↵
      2. Ivers N ,
      3. Jamtvedt G ,
      4. Flottorp S , et al
      . Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.doi:10.1002/14651858.CD000259.pub3
      OpenUrlPubMed
    41. 41.↵
      2. Camm AJ ,
      3. Lip GY ,
      4. De Caterina R , et al
      . 2012 focused update of the ESC guidelines for the management of atrial fibrillation: an update of the 2010 ESC guidelines for the management of atrial fibrillation. developed with the special contribution of the European Heart Rhythm Association. Eur Heart J. ;2012 33:21–2719 47 .doi:10.1093/eurheartj/ehs253
      OpenUrlCrossRefPubMedWeb of Science
    42. 42.↵
      2. Hodgkinson S ,
      3. Pollit V ,
      4. Sharpin C , et al
      . National Institute for Health and Care Excellence (NICE) Guideline Development Group. early management of head injury: summary of updated NICE guidance. BMJ 2014;348:g104.
      OpenUrlFREE Full Text
    43. 43.↵
      2. Konstantinides SV ,
      3. Torbicki A ,
      4. Agnelli G , et al
      . ESC guidelines on the diagnosis and management of acute pulmonary embolism. Eur Heart J 2014;35:3033–69–69.
      OpenUrl
    44. 44.↵
      2. Sié P ,
      3. Samama CM ,
      4. Godier A , et al
      . Surgery and invasive procedures in patients on long-term treatment with direct oral anticoagulants: thrombin or factor-Xa inhibitors. Recommendations of the Working Group on Perioperative Haemostasis and the French Study Group on Thrombosis and Haemostasis. Arch Cardiovasc Dis. 2011;104 (12):669–76.doi:10.1016/j.acvd.2011.09.001
      OpenUrlCrossRefPubMed
    45. 45.↵
      2. Vahanian A ,
      3. Alfieri O ,
      4. Andreotti F , et al
      . Guidelines on the management of valvular heart disease (version 2012). Eur Heart J. 2012;33 (19):2451–96.doi:10.1093/eurheartj/ehs109
      OpenUrlCrossRefPubMedWeb of Science
    46. 46.↵
      2. Murphy SN ,
      3. Mendis M ,
      4. Hackett K , et al
      . Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside. AMIA Annu Symp Proc. 2007:548–52.
    47. 47.↵
      2. Nkomo VT ,
      3. Gardin JM ,
      4. Skelton TN , et al
      . Burden of valvular heart diseases: a population-based study. Lancet. 2006;368 (9540):1005–11.doi:10.1016/S0140-6736(06)69208-8
      OpenUrlCrossRefPubMedWeb of Science
    48. 48.↵
      2. Iung B ,
      3. Baron G ,
      4. Butchart EG , et al
      . A prospective survey of patients with valvular heart disease in Europe: the Euro Heart survey on valvular Heart Disease. Eur Heart J. 2003;24 (13):1231–43.doi:10.1016/S0195-668X(03)00201-X
      OpenUrlCrossRefPubMedWeb of Science
    49. 49.↵
      2. Marcucci M ,
      3. Iorio A ,
      4. Nobili A , et al
      . Factors affecting adherence to guidelines for antithrombotic therapy in elderly patients with atrial fibrillation admitted to internal medicine wards. Eur J Intern Med. 2010;21 (6):516–23.doi:10.1016/j.ejim.2010.07.014
      OpenUrlCrossRefPubMed
    50. 50.↵
      2. Garber JL ,
      3. Willenborg KL ,
      4. Rose AE
      . Analysis of anticoagulant prescribing in non-valvular atrial fibrillation and development of a clinical tool for guiding anticoagulant selection. J Thromb Thrombolysis. 2015;40 (2):248–54.doi:10.1007/s11239-015-1223-5
      OpenUrl
    51. 51.↵
      2. Berti D ,
      3. Moors E ,
      4. Moons P , et al
      . Prevalence and antithrombotic management of atrial fibrillation in hospitalised patients. Heart 2015;101:884–93.doi:10.1136/heartjnl-2014-307059
      OpenUrlAbstract/FREE Full Text
    52. 52.↵
      2. Pereira de Sousa L ,
      3. Burba I ,
      4. Ruperto C , et al
      . Vitamin K antagonists in patients with nonvalvular atrial fibrillation: appropriateness and quality of treatment in an Italian cohort. J Cardiovasc Med. 2013;14 (7):534–40.
      OpenUrl
    53. 53.↵
      2. Kristensen SD ,
      3. Knuuti J ,
      4. Saraste A , et al
      . 2014 ESC/ESA guidelines on non-cardiac surgery: cardiovascular assessment and management: the Joint Task Force on non-cardiac surgery: cardiovascular assessment and management of the European Society of Cardiology (ESC) and the European Society of Anaesthesiology (ESA). Eur Heart J. 2014;35 (35):2014; 2383–431.
      OpenUrl
    54. 54.↵
      2. Boulkedid R ,
      3. Abdoul H ,
      4. Loustau M , et al
      . Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011;6 (6):e20476.doi:10.1371/journal.pone.0020476
    55. 55.↵
      2. Jones J ,
      3. Hunter D
      . Consensus methods for medical and health services research. BMJ. 1995;311 (7001):376–80.
      OpenUrlFREE Full Text
    56. 56.↵
      2. Falissard B
      . Mesurer la subjectivité en santé. Perspective méthodologique et statistique. Paris: Masson, 2001:109 p.
    57. 57.↵
      2. Uzuner O ,
      3. Solti I ,
      4. Cadag E
      . Extracting medication information from clinical text. J Am Med Inform Assoc. 2010;17 (5):514–8.doi:10.1136/jamia.2010.003947
      OpenUrlCrossRefPubMed
    58. 58.↵
      2. Hamon T ,
      3. Grabar N
      . Linguistic approach for identification of medication names and related information in clinical narratives. J Am Med Inform Assoc 2010;17 (5):549–54.doi:10.1136/jamia.2010.004036
      OpenUrlCrossRefPubMed
    59. 59.↵
      2. Pepe MS.
      The statistical evaluation of medical tests for classification and prediction. Oxford: Oxford University Press, 2003:302 p.
    60. 60.↵
      2. Bossuyt PM ,
      3. Reitsma JB ,
      4. Bruns DE , et al
      . The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem. 2003;49 (1):7–18.doi:10.1373/49.1.7
      OpenUrlAbstract/FREE Full Text
    61. 61.↵
      2. Whiting P ,
      3. Rutjes AWS ,
      4. Dinnes J , et al
      . Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess. 2004;8 (25):1–234.doi:10.3310/hta8250
      OpenUrlPubMed
    62. 62.↵
      2. Lucas NP ,
      3. Macaskill P ,
      4. Irwig L , et al
      . The development of a quality appraisal tool for studies of diagnostic reliability (QAREL). J Clin Epidemiol. 2010;63 (8):854–61.doi:10.1016/j.jclinepi.2009.10.002
      OpenUrlCrossRefPubMed
    63. 63.↵
      2. Lemeshow S ,
      3. Taber S
      . Lot quality assurance sampling: single- and double-sampling plans. World Health Stat Q. 1991;44 (3):115–32.
      OpenUrlPubMed
    64. 64.↵
      2. Jutand M ,
      3. Salamon R
      . [Lot quality assurance sampling: methods and applications in public health]. Rev Epidemiol Sante Publique. 2000;48 (4):401–8.
      OpenUrlPubMedWeb of Science
    65. 65.↵
      2. Delamarre D ,
      3. Bouzille G ,
      4. Dalleau K , et al
      . Semantic integration of medication data into the EHOP clinical data Warehouse. Stud Health Technol Inform. 2015;210:702–6.
      OpenUrl
    66. 66.↵
      2. Petit-Monéger A ,
      3. Saillour-Glénisson F ,
      4. Nouette-Gaulain K , et al
      . Comparing Graphical Formats for Feedback of Clinical Practice Data. A Multicenter Study among Anesthesiologists in France. Methods Inf Med. 2017;56 (1):28–36.doi:10.3414/ME15-01-0163
      OpenUrl
    67. 67.↵
      2. Campbell SM ,
      3. Braspenning J ,
      4. Hutchinson A , et al
      . Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11 (4):358–64.doi:10.1136/qhc.11.4.358
      OpenUrlAbstract/FREE Full Text

    Footnotes

    • Contributors Conceiving the study: APM, FT, VJ, PN, DB, MK, LRS, FSG. Design of the study: APM, FT, VJ, PN, DB, MK, RS, LRS, FSG. Writing the protocol: APM, FT, VJ, PN, DB, MK, RS, LRS, FSG. All authors read and approved the final manuscript.

    • Funding This work was supported by the French Ministry of Health, Research Program on the French healthcare system performance 2015 (PACHA study, grant number: PREPS 15 0433) without having any role in the study design, collection, analysis and interpretation of data, writing of the report, nor in the decision to submit the article for publication.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Collaborators PACHA study steering committee: Dr Berdaï Driss (Bordeaux University Hospital, Bordeaux, France); Pr Burgun Anita (Georges Pompidou European Hospital, Paris, France); Pr Cuggia Marc (Rennes University Hospital, Rennes, France); Grabar Natalia (University of Lille, Lille, France); Dr Jouhet Vianney (Bordeaux University Hospital, Bordeaux, France); Mrs Kret Marion (Bordeaux University Hospital, Bordeaux, France); Dr Noize Pernelle (Bordeaux University Hospital, Bordeaux, France); Dr Petit-Monéger Aurélie (Bordeaux University Hospital, Bordeaux, France); Dr Saillour-Glénisson Florence (Bordeaux University Hospital, Bordeaux, France); Pr Salmi Louis-Rachid (Bordeaux University Hospital, Bordeaux, France); Mr Sitta Rémi (Bordeaux University Hospital, Bordeaux, France); Dr Thiessard Frantz (Bordeaux University Hospital, Bordeaux, France). PACHA study scientific committee: Dr Berdot Sarah (Georges Pompidou European Hospital, Paris, France); Pr Cholley Bernard (Georges Pompidou European Hospital, Paris, France); Pr Coste Pierre (Bordeaux University Hospital, Bordeaux, France); Dr Escudié Jean-Baptiste (Georges Pompidou European Hospital, Paris, France); Pr Fischer Anne-Marie (Georges Pompidou European Hospital, Paris, France); Dr Galloula Alexandre (Georges Pompidou European Hospital, Paris, France); Pr Gaussem Pascale (Georges Pompidou European Hospital, Paris, France); Mr Diallo Gayo (Bordeaux University, Bordeaux, France); Dr Gilleron Véronique (Bordeaux University Hospital, Bordeaux, France); Dr Grellet Jean (Bordeaux University Hospital, Bordeaux, France); Pr Juvin Philippe (Georges Pompidou European Hospital, Paris, France); Dr Lafargue Aurélie (Bordeaux University Hospital, France); Pr Loriot Marie-Anne (Georges Pompidou European Hospital, Paris, France); Dr Massot Julien (Georges Pompidou European Hospital, Paris, France); Dr Mauge Laetitia (Georges Pompidou European Hospital, Paris, France); Mrs Mougin Fleur (Bordeaux University, Bordeaux, France); Pr Nouette-Gaulain Karine (Bordeaux University Hospital, Bordeaux, France); Dr Puymirat Etienne (Georges Pompidou European Hospital, Paris, France); Mr Rance Bastien (Georges Pompidou European Hospital, Paris, France); Dr Rouanet François (Bordeaux University Hospital, Bordeaux, France); Dr Sabatier Brigitte (Georges Pompidou European Hospital, Paris, France); Pr Saint-Jean Olivier (Georges Pompidou European Hospital, Paris, France); Dr Siguret Virginie (Lariboisiere Hospital, Paris, France); Dr Valdenaire Guillaume (Bordeaux University Hospital, Bordeaux, France).