You are here

Article Text

Article menu

Download PDFPDF

XML
Epidemiology
Research
Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies
  1. George A Neyarapally,
  2. Tarek A Hammad,
  3. Simone P Pinheiro,
  4. Solomon Iyasu
  1. Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland, USA
  1. Correspondence to Dr George A Neyarapally; george.neyarapally{at}fda.hhs.gov

Abstract

Objectives Pharmacoepidemiological studies are an important hypothesis-testing tool in the evaluation of postmarketing drug safety. Despite the potential to produce robust value-added data, interpretation of findings can be hindered due to well-recognised methodological limitations of these studies. Therefore, assessment of their quality is essential to evaluating their credibility. The objective of this review was to evaluate the suitability and relevance of available tools for the assessment of pharmacoepidemiological safety studies.

Design We created an a priori assessment framework consisting of reporting elements (REs) and quality assessment attributes (QAAs). A comprehensive literature search identified distinct assessment tools and the prespecified elements and attributes were evaluated.

Primary and secondary outcome measures The primary outcome measure was the percentage representation of each domain, RE and QAA for the quality assessment tools.

Results A total of 61 tools were reviewed. Most tools were not designed to evaluate pharmacoepidemiological safety studies. More than 50% of the reviewed tools considered REs under the research aims, analytical approach, outcome definition and ascertainment, study population and exposure definition and ascertainment domains. REs under the discussion and interpretation, results and study team domains were considered in less than 40% of the tools. Except for the data source domain, quality attributes were considered in less than 50% of the tools.

Conclusions Many tools failed to include critical assessment elements relevant to observational pharmacoepidemiological safety studies and did not distinguish between REs and QAAs. Further, there is a lack of considerations on the relative weights of different domains and elements. The development of a quality assessment tool would facilitate consistent, objective and evidence-based assessments of pharmacoepidemiological safety studies.

  • Epidemiology
  • Therapeutics

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

https://doi.org/10.1136/bmjopen-2012-001362

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Article focus

    • This article reviews the suitability and relevance of available tools for the assessment of the quality of pharmacoepidemiological safety studies.

    Key messages

    • In the context of regulatory safety-related decision making, quality assessment (ie, assessment of the risk of bias), informs the evaluation of available evidence and enhances the appropriate utilisation of available evidence in assessing the balance between benefits and risks of drugs.

    • The development of a consolidated reporting and quality assessment tool would enhance the consistent, transparent and objective evaluation of pharmacoepidemiological safety studies. If a tool is developed, it is important to determine if there is a need for tools tailored for specific study designs or if one tool that consolidates these considerations might be helpful.

    • Key findings from our review of quality assessment tools include:

      • Many available quality assessment tools do not include critical assessment elements that are specifically relevant to pharmacoepidemiological safety studies.

      • Most tools do not distinguish between reporting elements (REs) and quality assessment attributes (QAAs).

      • There is a lack of reported considerations on the relative weights to assign to different domains and elements with respect to assessing the quality of these studies.

    Strengths and limitations of this study

    • Strengths

      • A priori creation of a pharmacoepidemiological safety study assessment framework.

      • Comprehensive review of the literature.

      • Importance for safety-related regulatory decision making.

      • Potential to leverage other comparable efforts in the comparative effectiveness research arena.

    • Limitations

      • The purpose and scope of the reviewed tools varied greatly.

      • Each tool was reviewed by one reviewer.

    Introduction

    Several sources of evidence on drug safety issues inform Food and Drug Administration (FDA) postmarketing safety-related regulatory decisions, including spontaneous case reports, registries, observational pharmacoepidemiological studies, randomised controlled trials (RCTs), meta-analyses and other sources. Despite the well-known strengths of RCTs in the assessment of drug efficacy, specific issues related to the design, methodology and transparency of experimental studies may limit their ability to fully characterise the safety profile of drugs after marketing approval.1–4 Pharmacoepidemiological studies, typically observational in nature, represent an important hypothesis-testing mechanism in the evaluation of drug safety issues suspected at the time of approval and for new signals emerging in the postmarket period. In contrast to RCTs, such studies, which typically employ broader inclusion criteria and leverage claims or electronic medical record data, might better reflect the real-life experience of patients. Furthermore, pharmacoepidemiological studies afford the ability to investigate rare drug-related adverse effects, examine risks in patient subpopulations, and assess long-term adverse events. Recent health-related legislation will increase the availability and adoption of electronic healthcare data for such studies.5 ,6

    Despite the potential of pharmacoepidemiological safety studies to produce robust value-added data, the interpretation of findings from such studies can sometimes be challenging because of their well-recognised methodological limitations, including various sources of bias and confounding.7 These limitations also apply to the increasing number of comparative effectiveness epidemiological studies.8–10 The Institute of Medicine's recently published report highlights the importance of evaluating the quality of evidence and the significant scientific disagreements that have ensued over the quality of studies.11 Therefore, assessment of the quality of individual studies is essential to evaluating their credibility. Transparency in reporting on the design, conduct, analysis and results of these studies is a prerequisite for the assessment of the quality of the evidence; it is first necessary to understand the relevant aspects of the study design, conduct and analysis, along with the underlying assumptions and rationale behind the key scientific decisions undertaken by the study team to adequately evaluate the credibility of the study.12

    The FDA recently published a draft guidance on the design, conduct and reporting of pharmacoepidemiological safety studies using electronic healthcare data that is designed both to enhance the transparency of reporting of such studies and to encourage critical scientific thinking regarding their design and conduct.13 In the future, this guidance may improve the credibility of submitted pharmacoepidemiological studies by shedding light on the pertinent aspects of studies needed to inform the evaluation of the internal and external validity of their findings. However, even if the bar for transparency and reporting of these studies is raised, it will still be necessary to evaluate the contribution of these studies to the available evidence on an emerging drug safety issue. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) evidence assessment framework, used by clinical guideline developers, appropriately separates the initial processes of quality assessment and the weighing of evidence in the formulation of guideline recommendations.14 ,15 In the context of regulatory decision making concerning safety issues, the use of quality assessment tools to assess the risk of bias may add a measure of objectivity to the scientific judgment of the available evidence and improve the quality of decision making. The benefits may not only extend to improving decision making by regulators but also by journal editors and researchers as well as potentially improving the quality of performed studies by stimulating consideration of key aspects of these studies during the development of the study approach and protocol.

    Although many checklists and scales for the assessment of epidemiological studies exist,16–18 most are not specifically designed to evaluate pharmacoepidemiological safety studies. Importantly, although the principles of epidemiology apply across different fields, there are unique challenges in the design, conduct and evaluation of epidemiological studies of unintended drug harms that warrant consideration of developing a specific validated assessment tool (eg, confounding by indication is an important challenge that is unique to epidemiological studies of drug effects). Recent articles have suggested the need to develop tools for assessing the quality of these studies.19–22 A recent publication found that systematic reviewers and meta-analysts are misusing reporting tools like Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) due to the dearth of validated assessment instruments.23

    The main objective of this study is to critically evaluate the suitability and relevance of available tools for the assessment of pharmacoepidemiological safety studies. The ultimate goal is to stimulate discussion in the scientific community about the need for specific tools to facilitate the transparent, objective and consistent evaluation of study quality to inform safety-related regulatory decision making.

    Methods

    For purposes of this paper, quality assessment tools are defined as qualitative checklists and/or quantitative scales designed to facilitate assessment of the quality of epidemiological studies.

    A priori quality assessment framework

    To examine the utility of individual quality assessment instruments for the evaluation of pharmacoepidemiological safety studies, we created an a priori assessment framework, consisting of domains that include reporting elements (REs), and quality assessment attributes (QAAs) (table 1). Based on the expert opinion of FDA epidemiologists, concepts drawn from the FDA draft guidance on such studies, and key findings from seminal reviews tools and documents,13 ,24–28 we established the domains pertaining to the design, conduct and analysis of pharmacoepidemiological safety studies. Within each domain we listed critical elements that need to be considered for assessing the validity and interpretation of findings from such studies. We made a distinction between the REs and QAAs for each domain. This is an important distinction as some guidelines are strictly developed to discern and evaluate reporting whereas other tools are developed to evaluate quality, which requires assessment of reporting. The selected elements and attributes presented in this table are not intended to represent an all-inclusive list of factors, but rather to represent critical aspects impacting the internal and external validity of pharmacoepidemiological safety studies. Of note, although the QAAs necessarily involve some subjectivity, their inclusion in an assessment tool would facilitate the consistent and objective consideration and evaluation of key quality attributes across individual studies. As the GRADE framework developers have emphasised, although quality assessment is fundamentally subjective,29 developing a transparent, consistent approach to assessment of quality is important, especially in the regulatory and clinical arena as patients, healthcare professionals and sponsors benefit from consistent and transparent assessment of available evidence for use in decision making.

    View this table:
    Table 1

    Reporting elements (REs) and quality assessment attributes (QAAs) according to selected domains and percent representation among reviewed tools

    Literature search

    A comprehensive literature search of quality assessment checklists and scales was performed in MEDLINE, EMBASE and Web of Science. Search terms included: ‘assessment’, ‘tools’, ‘quality’, ‘medical research’, ‘evidence based research’, ‘evidence based medicine’, ‘meta-analysis’, ‘randomised controlled trials’, ‘biological product’, ‘drug’, ‘pharmaceutical preparation’, ‘biological therapy’, ‘bias’ and ‘epidemiology’. A total of 54 references were retrieved from this search. Two independent reviewers identified potentially relevant abstracts (n=26) from the initial literature review (inter-rater reliability >0.85). Inclusion criteria included quality assessment tools or relevant reviews developed to evaluate RCTs, observational studies or meta-analyses. Exclusion criteria consisted of clinical assessment tools, general articles or guidance on quality assessment and instruments or reviews focused strictly on reporting and not addressing quality assessment. After reviewing each paper, 10 relevant tools and 3 seminal reviews were identified; 13 were excluded based on the exclusion criteria above.

    The most recent, relevant review articles and some individual tools for assessing the quality of epidemiological studies were identified.25–27 The 2007 Sanderson review,25 the most recent, comprehensive review of instruments for assessing quality of epidemiological studies, served as the starting point.

    We also performed Google Scholar searches to identify relevant tools that might not be captured in the aforementioned search strategies. Google searches based on the first 50 hits included the following terms: ‘tool quality bias’; ‘quality assessment’; ‘pharmacoepidemiology’; ‘quality assessment epidemiology’; ‘tool quality assessment study’; and ‘scale quality assessment observational studies’. Furthermore, we identified and included an European Medicines Agency (EMA) methodological checklist24 because, although it is a reporting checklist, it includes domains and considerations designed to inform safety evaluations made at a drug regulatory agency.

    The assessment tools identified were reviewed by investigators based on a priori assessment criteria shown in table 1. The percentage of tools assessing the prespecified elements and attributes within domains (percent representation) was tabulated. During our review, we documented which tools employed some method of validation.

    Results

    Overall, out of 104 tools identified, a total of 96 distinct assessment tools, including 82 from the Sanderson review, 6 from the initial literature review, 7 from the Google search and 1 regulatory checklist (ENCEPP checklist) were considered for review (figure 1). Out of these, 61 were selected for the in-depth review.14 ,24 ,30–87 Tools exclusively focused on RCTs, tools focused on clinical assessments, and tools that did not include an explicit assessment framework were excluded (n=35).

    Figure 1
    Figure 1

    Percent representation by domain of reporting and quality aspects considered by the assessment tools.

    Representation of a priori assessment domains and elements within tools

    The proportion of reviewed tools that included REs and QAAs according to each a priori defined domain within the framework is shown in figure 1. Table 1 depicts the detailed results of our review of the domains, elements and attributes. We highlighted the representation of select RE and QAA under each domain that may have important implications for the assessment of a pharmacoepidemiological safety study. RE and QAA related to research aims were addressed in 69% (42/61) and 34% (21/61) of the tools, respectively. Regarding the domain assessing study population and data sources, 84% (51/61) of the tools included RE and 57% (35/61) included QAA (table 1).

    In all, 61% (37/61) of the tools included RE and 31% (19/61) included QAA under the exposure definition and ascertainment domain (table 1). With respect to outcome definition and ascertainment domain, 69% (42/61) of the tools included RE and 36% (22/61) included QAA (table 1). Out of the 61 reviewed tools, 85% (52/61) and 49% (30/61) included RE and QAA under the analytic approach domain (table 1). Under the results domain, only 36% (22/61) and 7% (4/61) of tools included RE and QAA, respectively (table 1).

    Of the 61 reviewed tools, 36% (22/61) and 20% (12/61) of tools included RE and QAA under the discussion and interpretation domain (table 1). Overall, 7% (4/61) of the tools addressed the description of the study team (RE) and the independence of team and funding sources (QAA).

    More than half of the reviewed instruments considered REs for domains including research aims, analytical approach, outcome definition and ascertainment, study population and exposure definition and ascertainment. Domains related to the discussion and interpretation, results and study team were considered in less than 40% of the reviewed tools. With the exception of the study population/data sources domain, QAA research domains were considered in less than half of the assessment tools, with less than 10% considering results and study team domains. Many did not address all pertinent domains.

    Most reviewed checklists and scales were not specifically designed to assess epidemiological studies of drug-related harms. Although the EMA framework was designed to increase transparency of pharmacoepidemiological studies, it focuses on reporting versus quality assessment. Our review constitutes the only recent comprehensive review of available assessment tools to determine if any are appropriate and sufficient for the evaluation of pharmacoepidemiological studies. Only a small number of the reviewed instruments employed some method of validation.30–36 Most of the tools did not differentiate between REs and QAAs whereas others stratified by these aspects. A small number included distinct assessment criteria for different epidemiological study designs (eg, case−control, cohort). Tools focused more on RE than QAA. Figure 1 displays the percentage of checklists and scales that included criteria on the assessment domains and elements.

    The proportion of reviewed tools that included REs and QAAs according to each a priori defined domain within the framework is shown in figure 1.

    Discussion

    Based on our review, there is no specific tool that is adequately designed for the robust evaluation of pharmacoepidemiological studies of drug safety. No single tool considered all the selected domains and elements and most tools failed to address critical evaluation elements. Making a distinction between RE and QAA is important as even if an element of the study is mentioned in the final report, one must then determine if this was appropriate for the specific study in the context of the drug safety question. Only a few instruments specifically made this distinction as we did in our a priori framework. Additionally, important RE and QAA were lacking in most of the checklists and scales we reviewed which highlights the need for a tool focused on the evaluation of epidemiological studies designed to evaluate drug-related harms; this need has been previously identified by others.88

    Quality attributes related to exposure definition and ascertainment were considered in less than half of the assessment tools, with less than 15% including RE and QAA pertaining to the comparator group, despite the fact that the selection of a comparator is critical for drug safety and effectiveness trials and epidemiological studies, as the choice of suboptimal comparators can provide misleading results.89 ,90 Only 30% of the instrument included quality assessment elements pertaining to the validity and appropriateness of the operational aspects of exposure ascertainment, and only 36% addressed quality attributes of validation of outcome ascertainment approaches. These are important facets of pharmacoepidemiological safety studies their misclassification may lead to false-negative findings regarding the association between a drug and adverse event.

    Only about 40% of the checklists and scales included QAA on approaches to handle confounding and biases. As observational studies are not randomised, the approaches to handle confounding and bias are of paramount importance.7 ,91 ,92 This is an important limitation of most tools because there are often uncertainties regarding results from pharmacoepidemiological studies due to the limitations of electronic healthcare data and complex nature of the practice of medicine.92 Only a small percentage of tools (28% RE; 18% QAA) included elements on the consideration of study findings in the context of the design, conduct, limitations and statistical power despite the fact that these elements are essential in assessing implications of study findings.

    Some of the tools we reviewed were designed as ‘all-purpose’ assessment instruments for evaluation of clinical trials and observational studies, while others focused on a particular study design (eg, case−control, cohort). It may be useful to create one consolidated, validated tool for evaluating observational pharmacoepidemiological safety studies focused on general reporting and quality attributes; tools for the specific study designs, that is, case−control and cohort studies, may also be useful due to some of the unique aspects of these designs. By creating such a tool, regulatory agencies, clinical guideline developers, and clinicians could consistently evaluate studies and for decision making. The creation of this instrument could be led by an independent group of expert methodologists, perhaps with input from multiple stakeholders, including regulators and professional organisations.

    Although we did not address weighing of importance of different domains and elements based on their relative impact on study contribution to the available streams of evidence, this may be an important consideration in the formulation of an assessment tool. Also, it is not clear if numerical scores are helpful in assessing the quality of epidemiological studies, as when numerical scores were used to evaluate systematic reviews or meta-analyses of such studies, they did not produce valid results.93 The appropriate tradeoff between the utility of a checklist or scale for review and the comprehensiveness of the evaluation elements has yet to be determined. This is complicated by the lack of validation of most of the available tools. Before these issues can be addressed, it is first necessary to engage in a broader discussion of the utility of such assessment tools in the evaluation of pharmacoepidemiological safety studies. It is worth noting that critical assessment elements of pharmacoepidemiological studies focused on effectiveness may be different than those focused on safety; however, pharmacoepidemiological comparative effectiveness studies focus on both comparative safety and benefits associated with drugs and thus such elements are not mutually exclusive.94 Thus, it is important to consider the potential to leverage current efforts to create a validated assessment tool (GRACE checklist95) for observational comparative effectiveness pharmacoepidemiological studies.

    Our review has some limitations. The purpose and scope of the checklists and scales we reviewed, varied greatly. Although we conducted a comprehensive review, there may be tools that we were unable to access or that were published after our search. If a reporting or quality assessment element contained some aspects of the element, we counted this as full representation, even if not all the important sub-elements were included. Each checklist or scale was reviewed by one study team member; repeating the evaluations via a second reviewer was deemed unnecessary at this stage as the primary goal of the review was to obtain a broad understanding of the utility of available assessment tools in evaluating pharmacoepidemiological safety studies based on a preliminary assessment framework. Some factors that may be increasingly relevant in future studies, such as electronic health records with linkages to other data sources like outpatient claims, health information exchanges or personal health records, were not included in our framework but may be included in a future validated instrument. Guidelines and checklists published after the time period of our review have included some elements that may be important for future linked studies which may leverage the increasing availability of these data sources.96

    In the evaluation of many emerging safety issues, pharmacoepidemiological safety studies are discussed and may influence safety-related decision making. However, often the quality-driven contribution of such studies is not discussed in a consistent way. The development of an assessment tool based on expert input may facilitate consistent, evidence-based quality assessment of such studies and the subsequent determination of their value based on evaluating the impact of bias on the robustness of a study results, and the interpretation of its findings, within the context of the specific drug safety issue. The framework we developed may serve as a foundation for future development of such an instrument. Efforts to improve the evaluation of the contribution of pharmacoepidemiological safety studies would be consistent with the FDA's focus on strengthening regulatory safety science.97 If after further consideration and discussions with stakeholders development of a tool to evaluate epidemiological data for drug safety is pursued, it would be necessary to first determine the scope of the assessment tool as well as steps for its comprehensive validation. Further, relevant aspects of the design and analysis of pharmacoepidemiology studies should be considered (we refer the reader to some helpful references).98–100 Importantly, such a tool would be intended to complement, and not replace, expert clinical, methodological and statistical expertise necessary to complete a robust evaluation and determination of the contribution of a specific pharmacoepidemiological safety study to the available evidence for regulatory decision making.

    Acknowledgments

    The authors would like to thank Gerald Dal Pan and Judy Staffa for their critical evaluation of the paper and thoughtful suggestions.

    Appendix: Flow diagram

    References

    1. ↵
      1. Hammad TA,
      2. Pinheiro SP,
      3. Neyarapally GA
      . Secondary use of randomized controlled trials to evaluate drug safety: a review of methodological considerations. Clin Trials 2011;8:559–70.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Ioannidis JP
      . Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Arch Intern Med 2009;169:1737–9.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Pitrou I,
      2. Boutron I,
      3. Ahmad N,
      4. et al
      . Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med 2009;169:1756–61.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Clark DW,
      2. Coulter DM,
      3. Besag FM
      . Randomized controlled trials and assessment of drug safety. Drug Saf 2008;31:1057–61.
      OpenUrlPubMedWeb of Science
    5. ↵
      American Recovery and Reinvestment Act. Pub.L. 2009;111–5
    6. ↵
      1. Bruen BK,
      2. Ku L,
      3. Burke MF,
      4. et al
      . More than four in five office-based physicians could qualify for federal electronic health record incentives. Health Aff (Millwood) 2011;30:472–80.
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Schneeweiss S,
      2. Avorn J
      . A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol 2005;58:323–37.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Dreyer N,
      2. Tunis S,
      3. Berger M
      . Why observational studies should be among the tools used in comparative effectiveness research. Health Affairs 2010;29:1818.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Dreyer N
      . Do we need good practice principles for observational comparative effectiveness research? GRACE Initiative (serial online) 2009. http://www.graceprinciples.org/art/GRACE_principles_25_october_2009.pdf (accessed on 3 Mar 2011).
    10. ↵
      1. Tunis SR,
      2. Benner J,
      3. McClellan M
      . Comparative effectiveness research: policy context, methods development and research infrastructure. Stat Med 2010;29:1963–76.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      Institute of Medicine. Ethical and Scientific Issues in Studying the Safety of Approved Drugs. 2012.
    12. ↵
      1. von EE,
      2. Altman DG,
      3. Egger M,
      4. et al
      . The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008;61:344–9.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      FDA. FDA Draft Guidance: Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data Sets. FDA Draft Guidance. 2011.
    14. ↵
      1. Atkins D,
      2. Best D,
      3. Briss PA,
      4. et al
      . Grading quality of evidence and strength of recommendations. BMJ 2004;328:1490.
      OpenUrlAbstract/FREE Full Text
    15. ↵
      1. Guyatt GH,
      2. Oxman AD,
      3. Vist GE,
      4. et al
      . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.
      OpenUrlFREE Full Text
    16. ↵
      1. Deeks JJ,
      2. Dinnes J,
      3. D'Amico R,
      4. et al
      . Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:173.
    17. ↵
      1. Sanderson S,
      2. Tatt ID,
      3. Higgins JP
      . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666–76.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Lohr KN
      . Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004;16:9–18.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Cornelius VR,
      2. Perrio MJ,
      3. Shakir SA,
      4. et al
      . Systematic reviews of adverse effects of drug interventions: a survey of their conduct and reporting quality. Pharmacoepidemiol Drug Saf 2009;18:1223–31.
      OpenUrlCrossRefPubMed
    20. ↵
      1. Chou R,
      2. Helfand M
      . Challenges in systematic reviews that assess treatment harms. Ann Intern Med 2005;142:1090–9.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Loke YK,
      2. Price D,
      3. Herxheimer A
      . Systematic reviews of adverse effects: framework for a structured approach. BMC Med Res Methodol 2007;7:32.
      OpenUrlCrossRefPubMed
    22. ↵
      1. Mallen C,
      2. Peat G,
      3. Croft P
      . Quality assessment of observational studies is not commonplace in systematic reviews. J Clin Epidemiol 2006;59:765–9.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. da Costa BR,
      2. Cevallos M,
      3. Altman DG,
      4. et al
      . Uses and misuses of STROBE statement: bibliographic study. BMJ Open 2011; 1:e000048 doi:10.1136/bmjopen-2010-000048.
    24. ↵
      EMA. Checklist for Methodological Standards for ENCePP Study Protocols. 2010.
    25. ↵
      1. Sanderson S,
      2. Tatt ID,
      3. Higgins JP
      . Tools for assessing the quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Deeks JJ,
      2. Dinnes J,
      3. D'Amico R
      . Evaluating non-randomised intervention studies. Health Technol Assess 2003;7:iii–x, 1–173.
    27. ↵
      1. Lohr K
      . Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004;16:9–18.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. von Elm E
      . STROBE. J Clin Epidemiol 2008;61:344–9.
    29. ↵
      1. Balshem H,
      2. Helfand M,
      3. Schunemann HJ,,
      4. et al
      . GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011;64:401–6.
      OpenUrl
    30. ↵
      1. Downs SH,
      2. Black N
      . The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377–84.
      OpenUrlAbstract
    31. ↵
      1. Macfarlane TV,
      2. Glenny AM,
      3. Worthington HV
      . Systematic review of population-based epidemiological studies of oro-facial pain. J Dent 2001;29:451–67.
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Margetts BM,
      2. Thompson RL,
      3. Key T,,
      4. et al
      . Development of a scoring system to judge the scientific quality of information from case−control and cohort studies of nutrition and disease. Nutr Cancer 1995;24:231–9.
      OpenUrlPubMedWeb of Science
    33. ↵
      1. Rangel SJ,
      2. Kelsey J,
      3. Colby CE,
      4. et al
      . Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. J Pediatr Surg 2003;38:390–6.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Genaidy AM,
      2. Lemasters GK,
      3. Lockey J,,
      4. et al
      . An epidemiological appraisal instrument—a tool for evaluation of epidemiological studies. Ergonomics 2007;50:920–60.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Santaguida P,
      2. Raina P
      . McMaster Quality Assessment Scale of Harms (McHarm) for primary studies. 2008 http://hiru.mcmaster.ca/epc/mcharm.pdf (accessed on 27 Oct 2010).
    36. ↵
      1. Cho MK,
      2. Bero LA
      . Instruments for assessing the quality of drug studies published in the medical literature. JAMA 1994;272:101–4.
      OpenUrlCrossRefPubMedWeb of Science
      1. Avis M
      . Reading research critically. II. An introduction to appraisal: assessing the evidence. J Clin Nurs 1994;3:271–7.
      OpenUrlPubMed
      1. Cameron I,
      2. Crotty M,
      3. Currie C,,
      4. et al
      . Geriatric rehabilitation following fractures in older people: a systematic review. Health Technol Assess 2000;4:111.
      1. Carneiro AV
      . Critical appraisal of prognostic evidence: practical rules. Rev Port Cardiol 2002;21:891–900.
      OpenUrlPubMed
    40. NHS. Critical appraisal skills programme: appraisal tools. Public Health Resource Unit N, editor. 2003.
    41. Center for Occupational and Environmental Health. Critical appraisal. School of Epidemiology and Health Sciences UoM, editor. 2003.
      1. DuRant RH
      . Checklist for the evaluation of research articles. J Adolesc Health 1994;15:4–8.
      OpenUrlCrossRefPubMedWeb of Science
      1. Elwood M
      . Forward projection—using critical appraisal in the design of studies. Int J Epidemiol 2002;31:1071–3.
      OpenUrlFREE Full Text
      1. Esdaile JM,
      2. Horwitz RI
      . Observational studies of cause−effect relationships: an analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis. J Chronic Dis 1986;39:841–52.
      OpenUrlCrossRefPubMedWeb of Science
      1. Gardner MJ,
      2. Machin D,
      3. Campbell MJ
      . Use of check lists in assessing the statistical content of medical studies. Br Med J (Clin Res Ed ) 1986;292:810–2.
      OpenUrlAbstract/FREE Full Text
    46. Health Evidence Bulletin. Questions to assist with the critical appraisal of an observational study eg cohort, case−control, cross-sectional. Wales, editor. 2004.
      1. Levine M,
      2. Walter S,
      3. Lee H,
      4. et al
      . Users’ guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. JAMA 1994;271:1615–19.
      OpenUrlCrossRefPubMedWeb of Science
      1. Lichtenstein MJ,
      2. Mulrow CD,
      3. Elwood PC
      . Guidelines for reading case−control studies. J Chronic Dis 1987;40:893–903.
      OpenUrlCrossRefPubMedWeb of Science
    49. Federal Focus I. The London principles for evaluating epidemiologic data in regulatory risk assessment. Fed Focus (serial online) 2004 http://www.fedfocus.org/science/london-principles.html (accessed on 5 Nov 2010).
      1. Margetts BM,
      2. Vorster V
      . Evidence-based nutrition—review of nutritional epidemiological studies. South Afr J Clin Nutr 2002;15:68–73.
      OpenUrl
    51. University of Montreal. Critical Appraisal Worksheet. 2004.
      1. Mulrow CD,
      2. Lichtenstein MJ
      . Blood glucose and diabetic retinopathy: a critical appraisal of new evidence. J Gen Intern Med 1986;1:73–7.
      OpenUrlCrossRefPubMedWeb of Science
      1. Whiting P,
      2. Rutjes AW,
      3. Reitsma JB,
      4. et al
      . The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 2003;3:25.
      OpenUrlCrossRefPubMed
    54. Scottish Intercollegiate Guidelines Network. SIGN 50: A guideline developer's handbook. Scottish Intercollegiate Guidelines Network (serial online) 2004.
      1. Parker S
      . Critical appraisal: Guidelines for the critical appraisal of a paper. Website (serial online) 2004.
      1. Zaza S,
      2. Wright-De Aguero LK,
      3. Briss PA,
      4. et al
      . Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. Am J Prev Med 2000;18:44–74.
      OpenUrlCrossRefPubMedWeb of Science
      1. Zola P,
      2. Volpe T,
      3. Castelli G,
      4. et al
      . Is the published literature a reliable guide for deciding between alternative treatments for patients with early cervical cancer? Int J Radiat Oncol Biol Phys 1989;16:785–97.
      OpenUrlPubMedWeb of Science
      1. Ariens GA,
      2. van MW,
      3. Bongers PM,
      4. et al
      . Physical risk factors for neck pain. Scand J Work Environ Health 2000;26:7–19.
      OpenUrlCrossRefPubMedWeb of Science
      1. Bhutta AT,
      2. Cleves MA,
      3. Casey PH,
      4. et al
      . Cognitive and behavioral outcomes of school-aged children who were born preterm: a meta-analysis. JAMA 2002;288:728–37.
      OpenUrlCrossRefPubMedWeb of Science
      1. Borghouts JA,
      2. Koes BW,
      3. Bouter LM
      . The clinical course and prognostic factors of non-specific neck pain: a systematic review. Pain 1998;77:1–13.
      OpenUrlCrossRefPubMedWeb of Science
      1. Carson CA,
      2. Fine MJ,
      3. Smith MA,
      4. et al
      . Quality of published reports of the prognosis of community-acquired pneumonia. J Gen Intern Med 1994;9:13–19.
      OpenUrlPubMedWeb of Science
      1. Loney PL,
      2. Chambers LW,
      3. Bennett KJ,
      4. et al
      . Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can 1998;19:170–6.
      OpenUrlPubMed
      1. Corrao G,
      2. Bagnardi V,
      3. Zambon A,
      4. et al
      . Exploring the dose−response relationship between alcohol consumption and the risk of several alcohol-related conditions: a meta-analysis. Addiction 1999;94:1551–73.
      OpenUrlCrossRefPubMedWeb of Science
      1. Garber BG,
      2. Hebert PC,
      3. Yelle JD,
      4. et al
      . Adult respiratory distress syndrome: a systemic overview of incidence and risk factors. Crit Care Med 1996;24:687–95.
      OpenUrlCrossRefPubMedWeb of Science
      1. Goodman SN,
      2. Berlin J,
      3. Fletcher SW,
      4. et al
      . Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Ann Intern Med 1994;121:11–21.
      OpenUrlCrossRefPubMedWeb of Science
      1. Jabbour M,
      2. Osmond MH,
      3. Klassen TP
      . Life support courses: are they effective? Ann Emerg Med 1996;28:690–8.
      OpenUrlCrossRefPubMedWeb of Science
      1. Kreulen CM,
      2. Creugers NH,
      3. Meijering AC
      . Meta-analysis of anterior veneer restorations in clinical studies. J Dent 1998;26:345–53.
      OpenUrlCrossRefPubMedWeb of Science
      1. Krogh CL
      . A checklist system for critical review of medical literature. Med Educ 1985;19:392–5.
      OpenUrlPubMedWeb of Science
      1. Littenberg B,
      2. Weinstein LP,
      3. McCarren M,
      4. et al
      . Closed fractures of the tibial shaft. A meta-analysis of three methods of treatment. J Bone Joint Surg Am 1998;80:174–83.
      OpenUrlPubMed
      1. Longnecker MP,
      2. Berlin JA,
      3. Orza MJ,
      4. et al
      . A meta-analysis of alcohol consumption in relation to risk of breast cancer. JAMA 1988;260:652–6.
      OpenUrlCrossRefPubMedWeb of Science
      1. Manchikanti L,
      2. Singh V,
      3. Vilims BD,
      4. et al
      . Medial branch neurotomy in management of chronic spinal pain: systematic review of the evidence. Pain Phys 2002;5:405–18.
      OpenUrl
      1. Meijer R,
      2. Ihnenfeldt DS,
      3. van LJ,
      4. et al
      . Prognostic factors in the subacute phase after stroke for the future residence after six months to one year. A systematic review of the literature. Clin Rehabil 2003;17:512–20.
      OpenUrlAbstract/FREE Full Text
      1. Nguyen QV,
      2. Bezemer PD,
      3. Habets L,
      4. et al
      . A systematic review of the relationship between overjet size and traumatic dental injuries. Eur J Orthod 1999;21:503–15.
      OpenUrlAbstract/FREE Full Text
      1. Reisch JS,
      2. Tyson JE,
      3. Mize SG
      . Aid to the evaluation of therapeutic studies. Pediatrics 1989;84:815–27.
      OpenUrlAbstract/FREE Full Text
      1. Stock SR
      . Workplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: a meta-analysis. Am J Ind Med 1991;19:87–107.
      OpenUrlPubMedWeb of Science
      1. van der Windt DA,
      2. Thomas E,
      3. Pope DP,
      4. et al
      . Occupational risk factors for shoulder pain: a systematic review. Occup Environ Med 2000;57:433–42.
      OpenUrlAbstract/FREE Full Text
      1. Bollini P,
      2. Garcia Rodriguez LA,
      3. Perez GS,
      4. et al
      . The impact of research quality and study design on epidemiologic estimates of the effect of nonsteroidal anti-inflammatory drugs on upper gastrointestinal tract disease. Arch Intern Med 1992;152:1289–95.
      OpenUrlCrossRefPubMedWeb of Science
      1. Ciliska D,
      2. Hayward S,
      3. Thomas H,
      4. et al
      . A systematic overview of the effectiveness of home visiting as a delivery strategy for public health nursing interventions. Can J Public Health 1996;87:193–8.
      OpenUrlPubMedWeb of Science
      1. Cowley DE
      . Prostheses for primary total hip replacement. A critical appraisal of the literature. Int J Technol Assess Health Care 1995;11:770–8.
      OpenUrlPubMedWeb of Science
    80. Effective Public Health Practice Project. Quality Assessment Tools for Quantitative Studies. 2003.
    81. School of Population Health. EPIQ (Effective Practice, Informatics, and Quality Improvement).  Faculty of Medical and Health Sciences. University of Auckland, 2004.
      1. Fowkes FG,
      2. Fulton PM
      . Critical appraisal of published research: introductory guidelines. BMJ 1991;302:1136–40.
      OpenUrlFREE Full Text
      1. Gyorkos TW,
      2. Tannenbaum TN,
      3. Abrahamowicz M,
      4. et al
      . An approach to the development of practice guidelines for community health interventions. Can J Public Health 1994;85(Suppl 1):S8–13..
      OpenUrl
      1. Spitzer WO,
      2. Lawrence V,
      3. Dales R,,
      4. et al
      . Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. Clin Invest Med 1990;13:17–42.
      OpenUrlPubMedWeb of Science
      1. Steinberg EP,
      2. Eknoyan G,
      3. Levin NW,
      4. et al
      . Methods used to evaluate the quality of evidence underlying the National Kidney Foundation-Dialysis Outcomes Quality Initiative Clinical Practice Guidelines: description, findings, and implications. Am J Kidney Dis 2000;36:1–11.
      OpenUrlPubMedWeb of Science
      1. El BN,
      2. Middel B,
      3. van Dijk JP,
      4. et al
      . Are the outcomes of clinical pathways evidence-based? A critical appraisal of clinical pathway evaluation research. J Eval Clin Pract 2007;13:920–9.
      OpenUrlPubMed
    87. ↵
      Salford University. Evaluation tool for quantitative research studies. 2001.
    88. ↵
      1. Chou R,
      2. Aronson N,
      3. Atkins D,
      4. et al
      . AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol 2010;63:502–12.
      OpenUrlCrossRefPubMedWeb of Science
    89. ↵
      1. Chokshi DA,
      2. Avorn J,
      3. Kesselheim AS
      . Designing comparative effectiveness research on prescription drugs: lessons from the clinical trial literature. Health Aff (Millwood) 2010;29:1842–8.
      OpenUrlAbstract/FREE Full Text
    90. ↵
      1. Psaty BM,
      2. Weiss NS
      . NSAID trials and the choice of comparators—questions of public health importance. N Engl J Med 2007;356:328–30.
      OpenUrlCrossRefPubMed
    91. ↵
      1. Strom B
      . Pharmacoepidemiology. 4th edn. Hoboken, New Jersey: John Wiley & Sons Ltd, 2005.
    92. ↵
      1. Brookhart MA,
      2. Sturmer T,
      3. Glynn RJ,
      4. et al
      . Confounding control in healthcare database research: challenges and potential approaches. Med Care 2010;48:S114–20.
      OpenUrlCrossRefPubMedWeb of Science
    93. ↵
      1. Stang A
      . Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol 2010;25:603–5.
      OpenUrlCrossRefPubMedWeb of Science
    94. ↵
      1. Conway PH,
      2. Clancy C
      . Charting a path from comparative effectiveness funding to improved patient-centered health care. JAMA 2010;303:985–6.
      OpenUrlCrossRefPubMed
    95. ↵
      GRACE. GRACE Checklist. 2012.
    96. ↵
      1. Hall GC,
      2. Sauer B,
      3. Bourke A,
      4. et al
      . Guidelines for good database selection and use in pharmacoepidemiology research. Pharmacoepidemiol Drug Saf 2012;21:1–10.
      OpenUrlPubMed
    97. ↵
      1. Hamburg MA
      . Shattuck lecture. Innovation, regulation, and the FDA. N Engl J Med 2010;363:2228–32.
      OpenUrlCrossRefPubMedWeb of Science
    98. ↵
      1. Miettinen OS,
      2. Caro JJ
      . Principles of nonexperimental assessment of excess risk, with special reference to adverse drug reactions. J Clin Epidemiol 1989;42:325–31.
      OpenUrlCrossRefPubMedWeb of Science
    99. ↵
      1. Guess HA
      . Behavior of the exposure odds ratio in a case−control study when the hazard function is not constant over time. J Clin Epidemiol 1989;42:1179–84.
      OpenUrlCrossRefPubMedWeb of Science
    100. ↵
      1. Schneeweiss S
      . A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiol Drug Saf 2010;19:858–68.
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Contributors AGAN was involved in Substantial contributions to the conception and design, acquisition of data or analysis and interpretation of the data; drafting the article or revising it critically for important intellectual content; and final approval of the version to be published.

    • TAH was involved in Substantial contributions to the conception and design, acquisition of data or analysis and interpretation of the data; drafting the article or revising it critically for important intellectual content; and Final approval of the version to be published.

    • SPP was involved in Substantial contributions to the conception and design, acquisition of data or analysis and interpretation of the data;

    • drafting the article or revising it critically for important intellectual content; and final approval of the version to be published. DSI was involved in Substantial contributions to the conception and design, acquisition of data, or analysis and interpretation of the data;

    • drafting the article or revising it critically for important intellectual content; and final approval of the version to be published.

    • Disclaimer The views expressed in this article are those of the authors and do not necessarily represent those of the Food and Drug Administration.

    • Competing interests None.

    • Ethics approval As this study involved a review of existing assessment tools, a formal ethics review was not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.