Article Text
PDF
Abstract
Objective To explore and categorise the state of existing literature for national programmes designed to affirm or establish the continuing competence of physicians.
Design Scoping review.
Data sources MEDLINE, ERIC, Sociological Abstracts, web/grey literature (2000–2014).
Selection Included when a record described a (1) national-level physician validation system, (2) recognised as a system for affirming competence and (3) reported relevant data.
Data extraction Using bibliographic software, title and abstracts were reviewed using an assessment matrix to ensure duplicate, paired screening. Dyads included both a methodologist and content expert on each assessment, reflective of evidence-informed best practices to decrease errors.
Results 45 reports were included. Publication dates ranged from 2002 to 2014 with the majority of publications occurring in the previous six years (n=35). Country of origin—defined as that of the primary author—included the USA (N=32), the UK (N=8), Canada (N=3), Kuwait (N=1) and Australia (N=1). Three broad themes emerged from this heterogeneous data set: contemporary national programmes, contextual factors and terminological consistency. Four national physician validation systems emerged from the data: the American Board of Medical Specialties Maintenance of Certification Program, the Federation of State Medical Boards Maintenance of Licensure Program, the Canadian Revalidation Program and the UK Revalidation Program. Three contextual factors emerged as stimuli for the implementation of national validation systems: medical regulation, quality of care and professional competence. Finally, great variation among the definitions of key terms was identified.
Conclusions There is an emerging literature focusing on national physician validation systems. Four major systems have been implemented in recent years and it is anticipated that more will follow. Much of this work is descriptive, and gaps exist for the extent to which systems build on current evidence or theory. Terminology is highly variable across programmes for validating physician competence and fitness for practice.
- EDUCATION & TRAINING (see Medical Education & Training)
- MEDICAL EDUCATION & TRAINING
- VALIDATING PHYSICIAN COMPETENCE
- SCOPING REVIEW
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- EDUCATION & TRAINING (see Medical Education & Training)
- MEDICAL EDUCATION & TRAINING
- VALIDATING PHYSICIAN COMPETENCE
- SCOPING REVIEW
Strengths and limitations of this study
Our research provides a rich description of differences, similarities and rhetorical complexities among global systems whose purpose is to ensure physician competence.
A limitation of our review is the reliance on publicly available information; intra-institutional documentation (eg, corporate minutes, technical reports) may have provided other important information.
We acknowledge that search results may be complicated by the lack of standardised language for describing and discussing validation systems as well as indexing terms within bibliographic databases. Terms such as revalidation, recertification, maintenance of competence, maintenance of licensure carry different connotations in different regions. Our findings provide a depth of inquiry around these terms that can be used for future research and scholarship.
Background
Internationally, there is agreement that initial certification obtained at the end of residency training is no longer adequate to sustain quality care across a physician’s career. The ‘once in, good for life’1 model has been successfully challenged by longitudinal studies showing that physicians benefit from well-structured educational programmes and approaches to their learning.2–4 Building on this work, national-level policy changes in several countries have aimed to validate the continuing competence of physicians in order to reduce physician errors, ensure quality care and reduce patient harm. What is less clear is the use of evidence or theoretical frameworks for informing the development of these large-scale, validation programmes.
Over the past two decades, several national validation systems have emerged, stimulated by increasing concerns related to variations in physician performance,5 adherence to healthcare indicators,6 quality of healthcare measures7 ,8 and growing public demands for transparency and accountability of health systems. These validation systems have been organised with similar end goals in mind, but display great variability in their approach to implementation. Some nations embrace the ‘up to date’ model of continuing medical education, where physicians are required to attend educational programmes and document their participation (eg attending continuing medical education events). Other nations have built systems that require physicians to demonstrate evidence of their continued competence and skill through audit and performance data. Traditionally, this has been a combination of both providing evidence of participation and demonstrating how participation has led to practice improvement or patient care outcomes.
Within this milieu, keywords such as revalidation, recertification, maintenance of certification and maintenance of licensure have been used to describe similar policies and procedures. We use the term ‘national physician validation systems’ as a broad label that includes all of these terms.
Not only does variation exist among the names and definitions of national physician validation systems, but a scarcity of evidence exists for the extent to which policies have been guided or informed by theory and/or evidence-informed decision-making. In an era when the effectiveness of systems for affirming physician competence is increasingly scrutinised,9–11 we sought to explore how countries developed and implemented national physician validation programmes.
Scoping reviews are categorised as exploratory projects intended to rapidly map the literature to identify key concepts, theories and sources of evidence. They are often undertaken when feasibility of the research is considered to be a challenge either because of presumed diversity, and/or it is thought that little literature exists.12 In particular, it can be undertaken when an area has not been comprehensively reviewed before. Given our nascent understanding of the scope or diversity of literature, and a recognition that to our knowledge no comprehensive review was available, we undertook a scoping review to summarise information for (1) what theoretical models or policy structures have been described as informing the development and/or implementation of national physician validation systems, (2) what national continuing professional development (CPD) programmes were used or developed to implement strategies/systems and (3) what specific contextual factors have been described as catalysts or influential in developing, shaping or changing these systems.
Methods
Using Arksey and O'Malley's12 scoping review methodology, our analysis included the necessary key stages of development including identifying research questions, locating relevant studies, defining study selection, charting data and collating and summarising the results. Search strategies were developed iteratively and in collaboration with a librarian experienced in the development of complex search strategies. The scoping review team used a combination of subject headings (MeSH) and free-text terms for the search to ensure breadth and depth of coverage in MEDLINE, Educational Resources Information Centre and Sociological Abstracts. The search was also approved through peer review using the Peer Review of Electronic Search Strategies checklist.13
Given the diverse nature of the information, we sought to summarise and to ensure comprehensiveness in our approaches, a thorough supplemental internet search for grey literature (eg, technical or white paper reports) was conducted. Relevant web resources were searched in a similar manner using a number of techniques: use of the search bar, keywords and snowballing—linking through to relevant context as identified. All search actions were recorded per site, and relevant records were catalogued as having been identified through non-traditional search techniques.
Selection
English-language reports of any study or publication type were eligible for inclusion. Reports were included for analysis if they (1) described a national-level system (jurisdictional or by specialty), (2) highlighted a system that was designed to affirm or establish the continuing competence of physicians (lifelong learning) and (3) reported relevant information (purpose of the programme or system, etc). Reports outside of human medicine (eg, veterinary sciences) and published prior to 1 January 2000 were excluded from our review.
A relevance assessment tool was developed collaboratively by the team and tested a priori to ensure consistency of assessment. The tool assessed four broad domains for inclusion specific to (1) population of interest, (2) scope or jurisdiction, (3) a system for affirming competence (making explicit the integration of the programme, tool, or strategy within a system of Maintenance of Certification, Revalidation or Recertification) and (4) relevant information (publication-level characteristics, terminology, theoretical model(s) or policy structure(s), contextual or other factors as important to or influential in what was developed or implemented, and use of evidence or data that informed or served as the basis for decision-making). All bibliographic and web-based search results were organised by a coordinator who removed duplicate records.
Using reference management software, the title and abstract of all records were provided to each reviewer using an assessment matrix to ensure independent, duplicate screening. Two reviewers independently reviewed and made decisions on titles and abstracts using an accelerated screening algorithm whereby a record could ‘pass’ level I when only one reviewer indicated it was potentially relevant. Alternatively, the title/abstract was excluded only when there was agreement by two reviewers (for exclusion). Reflective of evidence-informed best practices to decrease errors,14 each assessment matched an evidence synthesis methodologist as well as content-area expert on each assessment. In cases where conflicts could not be resolved between the primary researchers, another review author (CC) was consulted.
Charting
In a scoping review, charting is the process whereby reviewers synthesise and interpret qualitative data by sorting material according to key issues and themes.14 This process is similar to narrative reviews which examine the context of outcomes or processes included in a study in order to provide greater more nuanced information for reviewers.12 An approach akin to qualitative content analysis was used to analyse information from reports by the team. For example, definitions for each term (eg, revalidation) were extracted verbatim and the text reviewed with an aim of generating the simplest and most likely explanation for phenomena based in inferential reasoning and the generation of themes. For example, a predefined glossary of descriptors (recertification, maintenance of competence, revalidation, continuing professional development, maintenance of certification and maintenance of licensure) was used to ‘label’ records for charting purposes. Further, to provide standardisation across reviewers, study design was classified using definitions as published within the Cochrane Library Glossary of Terms.15
Data pertaining to demographic and study-level characteristics (publication year, publication type, study design, country, etc), terminology (use of terminology or nomenclature, verbatim definitions), system characteristics (when established, why established, mandate, attributes, processes, framework/theoretical underpinning, etc) and information thought to be relevant for responding to our questions were extracted (eg, contextual factors influencing programme development or design, etc).
Results
Searches resulted in retrieval of 7526 records for review following an exercise of de-duplication by the research coordinator. We excluded 7294 articles following title and abstract screening for lack of relevance. Full-text reports were obtained and reviewed by two authors (232) resulting in a total of 45 reports identified for final inclusion within our study (figure 1).
Flow diagram for paper selection process.
Publication dates ranged from 2002 to 2014 with the majority of publications occurring in the previous 6 years (35/45 publications >2008). Country of origin (defined as that of the primary author) included the USA (N=32), the UK (N=8), Canada (N=3), Kuwait (N=1) and Australia (N=1). Publications were published in the form of ‘journal articles’ (N=35) with the remaining 10 reports described/categorised as ‘general report(s)’ (N=5), ‘manual’ (N=1), ‘guideline’ (N=1), ‘descriptive review’ (N=1), technical paper (N=1) and an ‘other’ (N=1).
Reviewers were asked to categorise each report using the classification system, and decisions were reviewed for accuracy and quality assurance purposes. Reports were primarily descriptive defined as that which ‘… describes characteristics of a sample of individuals; unlike an experimental study, the investigators do not actively intervene to test a hypothesis, but merely describe the health status or characteristics of a sample from a defined population’.15
The frequency by which terms were used to describe the national systems included Maintenance of certification (N=22), Revalidation (N=9), Maintenance of licensure (N=6), Recertification (N=5), Maintenance of professional competence (N=1), Osteopathic continuous certification (N=2). Table 1 summarises the terminology descriptors succinctly. The majority of reports were descriptive summaries that either explained a system generally (eg, in a specialty journal for its readership) or the evolution of its history including the rationale for adopting a system for ‘affirming competence’. These descriptive papers generally outlined the components of the programme(s), and detailed the requirements expected within the national physician validation system (eg, documentation requirements, credits and in some cases expected ‘outcomes’).
Summary of national validation systems
National physician validation systems
The first question for our review was to answer the extent to which theoretical models or frameworks have been used to develop and/or implement national systems for physician validation. Four primary systems were identified (1) the maintenance of certification programme of the American Board of Medical Specialties (ABMS), (2) the maintenance of licensure programme of the American Federation of State Medical Boards, (3) the Canadian revalidation programme of the Federation of Medical Regulatory Authorities of Canada and (4) the UK revalidation programme of the General Medical Council (GMC).
Terminology
Terminology used to describe national physician validation systems is complicated. Our review summarises the variability in terms and how they are defined or described. A number of reports have highlighted the need for a process to compare systems and present definitions by country. However, our findings have been limited to national systems in the USA, the UK and Canada, and have ignored the terms recertification and revalidation.
Recertification is a term used in the USA for participation in the American Board of Medical Specialties Maintenance of Certification Process that ensures a commitment to lifelong learning and competency in the six core competencies of the Accreditation Council for Graduate Medical Education. Revalidation is used differently in the UK and Canada. In the UK, revalidation is a term that refers to both relicensing which is a requirement for all physicians, and recertification which is required by specialists. In Canada, revalidation refers to participation in nationally validated programmes of the national certifying Colleges (Royal College of Physicians and Surgeons of Canada and College of Family Physicians of Canada) designed to enhance the continuing professional development of individual specialties.16
Despite the small numbers of terms published to describe national physician validation programmes, there is considerable variability in the key descriptors used to define or explain the purpose of the programme. A standardised, or accepted definition for each term is not present within the literature. Below are common descriptors for each term identified within the documents included in our review (table 2). Other terms were represented in only one report and included maintenance of professional competence, osteopathic continuous certification, and five did not report a specific ‘term’.
Terminological descriptions
Contextual factors
Thirty-three of the 45 articles described important contextual factors that were influential in the development of national physician validation systems. We defined contextual factors as ecological, or environmental factors that were described as relating to the establishment or development of national systems. The majority of the contextual factors can be summarised within three primary constructs including quality of care, professional competence and medical regulation.
Quality of care
Numerous reports described ‘Quality of Care’ factors that influenced the development of physician validation programmes. In the USA, for example, the ABMS developed their maintenance of certification programme in response to increased public demands for accountability and transparency. These concerns were reported as stemming from public and professional concerns related to medical errors, patient safety and variation in quality of care. Two landmark reports ‘To Err Is Human’7 and the ‘Quality Chasm’23 were referenced specifically as influencing the design and development of the ABMS maintenance of certification process.24–34 Other articles noted the influence of the ‘quality of care movement’ and ‘patient safety movement’16 ,18 ,24 ,25 ,27 in general, or the ethical implications of the social contract between the profession and the public that requires the profession to ‘guard the public health’.35
Several American medical boards described the importance of restoring the ‘public trust’ or ‘protecting’ the public interest’36–39 as part of their mission to satisfy the public and all stakeholders that ‘specialists are competent to maintain their competency throughout the span of their professional careers’.40 The safety of the healthcare system and growing patient awareness and expectations about the quality of care, they should expect to receive were challenges to continuously improve physician performance and improve healthcare quality, or improve patient outcomes.18 ,20 ,25 ,31 ,33 ,37 ,39 ,41 ,42 These public expectations, coupled with the concerns related to the costs, safety and quality of healthcare created the perspective that the Recertification process is a ‘move to continuous assessment of physician quality’.37 Finally, several articles cited data from polls that identified public expectations that the profession engage in frequent review and testing.17 ,26 ,36
Professional competence
A number of reports highlight the importance of ensuring physician competence in light of the rapid advancement of clinical medicine in the knowledge economy. This body of information included reports whose orienting framework described physician knowledge, competence or performance deterioration over time. This trend was addressed, in part, by evidence that certification status had a higher correlation with reduced patient mortality, leading several authors to view certification as a means to enhance healthcare quality and protect the public.22 ,24 ,26 ,32 Furthermore, studies highlighted concerns regarding the variability of translating knowledge into evidence-informed decision-making. The significant growth in evidence that informs practice today requires that physicians engage in continuing education and assessment in order to promote reflection and ongoing quality improvement of the care they provide.26 ,35 ,36
Medical regulation
A number of reports elucidated that the establishment of validation programmes was significantly influenced by high-profile catastrophic incidents and public enquiries that, collectively, were viewed as a failure of medical regulation to provide necessary assurances that ‘physicians have the competence and ethical integrity to care for the public’.43 In the UK in particular, specific references to the Royal Bristol Infirmary and the Shipman inquiries43–46 focused on the absence of policies and processes for ensuring patient safety and physician efficacy.44
In 2002, the UK's GMC concluded that “physicians should undergo regular review of their performance to reassure the public and government that they remain fit to practice across their professional lifetime.” The subsequent development of the 2007 white paper ‘Trust, Assurance and Safety’ (Department of Health), and Sir Liam Donaldson's paper on ‘Good Doctors, Safer Patients’,47 contributed to the specifics of the Revalidation strategy that was implemented in December 2012.
Table 3 describes keywords or phrases that were most often associated with each system descriptor as well as the underlying educational focus, the role of assessment and the organisations responsible for each. Despite the variation in terminology, a set of common principles emerged and included: the demonstration of lifelong learning through engaging in continuing professional development (CPD) activities; continuously assessing competence and performance using multiple strategies and tools to improve the quality of care; and addressing public concerns regarding the need for greater transparency and accountability of the profession to protect the public and maintain the public's trust.
System attributes for revalidation and recertification
Discussion
The purpose of our scoping review was to comprehensively examine the body of literature for the development of, processes involved, and implementation of national systems intended to support and affirm physician competence. Our review has demonstrated the existence of a relatively small body of literature describing national systems governing programmes or strategies to sustaining physician competence. While certain principles appear to be stable concepts across all reports within our review, there is an array of terms and definitions that pervade the literature and point to important differences to inform interpretation. While relatively few terms were identified, it is clear that after nearly a decade of publication, variability in the application and use of the terms exists.
Our findings are echoed in a recent report by Archer and colleagues, that analysed prevailing definitions of ‘revalidation’. Using a discourse analysis approach, the research team analysed interviews with 31 leading medical and legal revalidation policymakers.48 Regulation and professionalism emerged as two ‘discourses’, often interchangeably using the term revalidation. In an era of increasingly globalised medical research and international collaboration, national systems for validating physician competence are potentially using competing ‘discourses’ for crucial terms such as revalidation. A key strategic future direction for medical education policymakers interested in international collaboration would be to highlight the implementation of a worldwide CPD nomenclature.
Although the aims or goals of national systems were frequently cited, no description of any theoretical framework guiding their establishment or the use of research evidence that informed or influenced their design was reported. One potential explanation for this could be that the governance and design of these national programmes rested with regulatory or governmental organisations whose primary aim was to respond to external demands for change. Another possible explanation is that study authors were from organisations that did not have ultimate responsibility for the national programme or framework but were simply expressing their requirements, implementation strategy and/or current or anticipated challenges with future implementation. Finally, the information may be resultant from insufficient reporting of this detail. It will be important to extend our work and seek to interview developers and those involved in the development of national systems and programmes.
One report described systems specifically for a ‘specialty area’ (Maintenance of certification in dermatology: what we know, what we don't), and cited a variety of studies to support the rationale or the need for Maintenance of certification and associated measures of quality of care.17 ,19 ,36 For example, Shaw et al16 examined national systems of three countries (Canada, the UK and the USA), and provided a summary of ongoing research related to the implementation of new methods and tools being developed, piloted and evaluated. We surmise that the notable absence of evidence cited in these papers did not necessarily imply that evidence or theoretical frameworks were not used to inform the development of national systems. Perhaps such sources instead were not accessible by our search methods. We recommend researchers interested in developing future studies relating to the establishment of national systems for physician revalidation consider that a wealth of information may be available within institutional and administrative repositories—for example, policy papers or minutes of meetings and extensive searching and communication directly with institutions may be required.
Our results also identified that each system shares a number of similar obligatory requirements. For example, all validation systems require physicians to engage in and report on their participation in CPD or learning activities relevant to their practice, whereas the espoused purpose of various assessment strategies was primarily educational or formative, aiming to promote or enhance learning, quality improvement or continuous enhancement of practice.
We identified that the role for assessment could shift from formative to summative if the assessment strategies identified either very poor performance, or where risks to patient safety were deemed to be high. Under such circumstances, further detailed assessments could be mandated with the potential for temporary withdrawal of or limitations on licensure until the identified deficiencies have been sufficiently and appropriately remediated. Only the Maintenance of Certification Program of the American Board of Medical Specialties requires each American Board to develop a high-stakes summative assessment of knowledge that all specialists must pass every 10 years to maintain their certification status.
Much of the research demonstrates that greater attention is being given to the role physician validation systems play in ensuring both quality care and promoting physician accountability in healthcare. As more countries decide to pursue validation, the best practices for structuring these systems will continue to be hotly contested and debated. How a nation determines it will handle physician validation has implications in terms of health policy, medical practice, patient care, costs, the culture of medical professionalism and how physicians situate themselves in terms of their own professional identity. To cite a current, relevant example, The Medical Board of Australia (MBA) has recently commissioned research to establish whether or not there is an evidence base for the validity of validation systems in comparable countries (http://www.medicalboard.gov.au/News/2015–03-24-media-release.aspx). The MBA cites evidence from Canada and the USA on gaps in clinical care that could be ameliorated through a validation system for physicians. The MBA commissioned a research group from the UK for their report, a group that was extensively involved in the UK's revalidation schema.
The establishment of a national physician validation system now also extends far beyond the borders of a single country. Our review did not identify reports addressing uniformity of accreditation standards or ways to ensure physicians can confidently train across borders. However, it is our belief that uniformity could ensure that physicians seeking educational activities not provided within their own country would be similarly equivalent with regard to educational components, outcomes and quality. To this end, mutual recognition could be developed for all CME accreditation systems, and would be required to be same, or ‘substantively equivalent’. Substantive equivalency is based on the ability of each system to reflect a common set of principles, values and metrics.49
Regardless of the scope of learning activities included within national CME frameworks of learning, or the number of credits expected to be achieved, the process of substantive equivalency should ultimately be constructed on the ability of CME systems to demonstrate how these principles, values and metrics are implemented and expressed within each system.
To this end, we also believe that international collaboration will play a key role in the establishment of national physician validation systems, which suggests that research on national physician validation systems will be increasingly valuable and relevant to policymakers worldwide.
Limitations
We did not include non-English language publications, although research has demonstrated that the evolution of validation systems is taking place internationally.50 Similarly, despite the existence of a national validation system in New Zealand, we did not identify a single report from that country that sufficiently met our inclusion criterion. Two reports did discuss or describe the New Zealand programme, but within the context of a global perspective.20 ,22 In other jurisdictions (eg, Medical Council of Singapore), there are plans for a system of revalidation.51 Evidence within biomedical reviews suggests that highly cited areas of inquiry, evidence syntheses are out of date within a relatively short period of time.52 In all, these gaps point to the evolving, international context of physician validation as an emerging area of enquiry for researchers.
Despite identifying 45 applicable reports through extensive searching, its likely additional data would only be available within the grey literature, which historically, is difficult to locate. Our review was based primarily on ‘published’ information, and likely excluded other jurisdictions and intra-institutional documentation that might have been acquired using other approaches.
Furthermore, we acknowledge that search results may be complicated by the lack of standardised language for describing and discussing validation systems. Terms such as revalidation, recertification, maintenance of competence, maintenance of licensure and others carry different connotations in different regions. We hope our findings provide a depth of inquiry around these terms that can be used for future research and scholarship.
Conclusion
This scoping review provides a comprehensive overview of the state of the literature pertaining to the implementation of national systems developed to affirm the continuing competence of physicians. This review has identified 45 reports that outline the complexity, synergies, terminology and intricacies inherent in given systems. We believe our findings describe the landscape of revalidation, recertification and maintenance of competence, along with the contexts within which these systems were developed. The primary focus of the review was ‘process-oriented’, and our findings suggest there is relatively little objective evidence for the ‘how and why’ of existing national systems for affirming the competence of physicians.
Acknowledgments
The authors thank Allan McDougall, Research Associate, Research Unit, Royal College of Physicians and Surgeons of Canada for his critical review of the manuscript.
References
Footnotes
Twitter Follow Tanya Horsley at @thorsley_handle
Contributors All authors listed on the manuscript contributed substantially to both the project and review of the manuscript. TH conceptualised the project, completed the initial draft of the protocol and manuscript, oversaw project conduct, screening, charting and interpretation, and is accountable for all information here. JL and CC contributed significantly to the review of both the protocol and final manuscript, and engaged in screening and charting and interpretation. FB reviewed the protocol, manuscript and completed charting. EC led the development of the search strategies, conducted developmental testing and contributed to review of the protocol and manuscript. She drafted the initial search methods. JZ coordinated the team and project and engaged in screening, conflict resolution, tracking, protocol and manuscript review.
Funding Royal College of Physicians and Surgeons of Canada.
Competing interests CC, TH and JZ are employees of the Royal College.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.