You are here

  • Home
  • Archive
  • Volume 9, Issue 2
  • Updating quality indicators for low-risk labour care in Japan using current clinical practice guidelines: a modified Delphi method

Article Text

Article menu

Download PDFPDF

XML
Obstetrics and gynaecology
Research
Updating quality indicators for low-risk labour care in Japan using current clinical practice guidelines: a modified Delphi method
  1. Kayo Ueda1,2,
  2. Misato Kaso1,
  3. Shosuke Ohtera1,
  4. Takeo Nakayama1
  1. 1 Department of Health Informatics in the School of Public Health, Kyoto University, Kyoto, Japan
  2. 2 Graduate School of Nursing Women’s Health & Midwifery, School of Medicine Faculty of Nursing Department of Maternal Nursing, Nara Medical University, Kashihara, Japan
  1. Correspondence to Dr Kayo Ueda; ueda.kayo.85a{at}st.kyoto-u.ac.jp

Abstract

Objectives Quality indicators are measurable elements widely used to assess the quality of care. They are often developed from the results of systematic reviews or clinical practice guidelines. These sources are regularly updated in line with new clinical evidence, but there are few articles on updating quality indicators based on clinical practice guidelines. This study aimed to update the quality indicators developed for low-risk labour care in Japan in 2012, mainly drawing on new or updated clinical practice guidelines, and making the process clearly visible and assessable.

Design and setting We used a modified Delphi method for the update. The procedure included four steps: (1) updating the definition of low-risk labour; (2) reviewing the literature published between June 2012 and December 2015 using five guidelines and two quality indicator databases to extract potential candidate indicators; (3) formation of a multidisciplinary panel including mothers and (4) panel ratings (two rounds between February and April 2016) on the validity of the candidate indicators, and judging the validity of the previous quality indicators drawing on the new evidence.

Participants A multidisciplinary panel of 13 clinicians, including obstetricians, paediatricians and midwives, plus 3 non-clinician mothers.

Results The literature review identified 276 new recommendations from 27 clinical practice guidelines including 2 published in Japan and 21 quality indicators. We developed 13 new candidate indicators from these sources and panel recommendations, 12 of which were approved by the multidisciplinary panel. The panel also accepted all 23 existing quality indicators as still valid, resulting in a total of 35 quality indicators for low-risk labour.

Conclusions We successfully updated the quality indicators for low-risk labour care in Japan. The procedure developed may be useful for updating other quality indicators based on new clinical practice guidelines.

  • quality indicator
  • clinical practice guidelines
  • update
  • modified delphi method
  • low-risk labor

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2018-023595

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Drawing on current clinical practice guidelines, we efficiently updated the quality indicators for low-risk pregnancy care using a modified Delphi method.

    • After clarifying the method for recruiting a multidisciplinary panel, our panel included 13 clinicians and 3 mothers who were non-clinicians.

    • The panel composition may have affected the consensus process of developing and updating quality indicators.

    • This update procedure may be useful for updating quality indicators based on revised or new clinical practice guidelines in other fields.

    Background   

    The US Institute of Medicine has defined the quality of care as ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’.1 Quality indicators are measurable elements that are widely used to assess the quality of care. They are usually drawn up from up-to-date published evidence including systematic reviews and clinical practice guidelines.2 3

    It has long been recognised as necessary to update clinical practice guidelines and systematic reviews as new relevant studies are published.4–7 This suggests that it is also important to update quality indicators based on clinical practice guidelines and systematic reviews,8–13 but few articles have ever reported on this updating process. If clinical practice guidelines make new recommendations, it is essential to discuss whether to incorporate these into existing quality indicators.4

    There are four reasons to update quality indicators. The first is because it is necessary to reassess the evidence, for example, because new clinical practice guidelines have been published.14–17 The second is during a review of concepts or definitions of expert terms, such as revision of definitions and metrics.18 19 The third is when adjusting the data collection system or method used by healthcare providers, such as updates to indicator content or clarifying and adjusting a particular procedure.14 19 The fourth is when there is a ceiling effect. We defined the ‘ceiling effect’ as a phenomenon in which the scores for the quality indicator are near their maximum value. If a ceiling effect occurs, it is impossible to increase the score significantly. It is, therefore, also very difficult to evaluate the quality improvement or detect differences between the measured score and other hospitals or over time.20 21 Very few previous reports have sufficiently described the methods and processes used to update quality indicators.6

    In Japan, quality indicators have rarely linked to clinical practice guidelines. There is no national data collection or reporting on patient safety in Japan, and this is not monitored nor shared across hospitals. The authors developed quality indicators for low-risk labour care in 2012. The annual number of live births in Japan is about 980 000, of which 92% are normal term infants. In 2016, the maternal mortality rate was 3.4 and the perinatal mortality rate was 3.6.22

    The majority involve low-risk labour,23 which seldom results in complications. We used clinical practice guidelines and quality indicators which were related to low-risk birth and published between July 2007 and June 2012. The results were published in English in 2017.24

    Quality indicators for low-risk labour care are necessary for several reasons. First, healthcare providers need to share a common understanding of checks that will keep birth-related risks low, reducing potential complications and problems caused by insufficient communication. Second, healthcare providers can compare the quality of care with other facilities, to improve their practice. Third, quality indicators can be used to ensure midwives’ accountability for low-risk labour care.

    More recently in Japan, there has been an increasingly uneven distribution of obstetricians, a decrease in facilities specialised for childbirth. It has become difficult for obstetricians to be responsible for all low-risk births in the hospital, so systems allowing midwife-led care are being examined. It is, therefore, even more important to guarantee the quality of care for low-risk labour.

    Since the development of the original quality indicators in 2012, new clinical practice guidelines on low-risk labour have been published in Japan, which suggests the need to reconsider the quality indicators.25–27 This study, therefore, aimed to update the quality indicators for low-risk labour care drawing on new clinical practice guidelines and to propose an efficient way to update quality indicators.

    Methods

    Indicator update

    Review of the literature to assess whether any updating was necessary

    We assessed whether the information in recently published clinical practice guidelines touching on low-risk labour care could affect the original quality indicators. We also made an abbreviated search of previously cited clinical practice guidelines to identify any new recommendations or information. Enough new information emerged that we decided that it was necessary to update the quality indicators for low-risk labour.

    The update process

    We used a modified Delphi method to integrate scientific evidence with expert panel opinions. This method has been widely used to develop quality indicators.2 6 28–33 To develop a consensus, panel members individually rated candidate indicators in two rounds, through a postal questionnaire, with the second round used to resolve any issues emerging from round 1. In the second round, the rating results and all comments, including agreement and disagreement from the first round, were shared anonymously with the panel members. Disagreements were discussed by email or telephone between a facilitator (a researcher: KU) and the panel member to assure anonymity of comments. We used the consensus-based reporting standards for guideline-based performance measure development and re-evaluation proposed by the Guidelines International Network Performance Measures Working Group.32

    Step 1: update of the definition of low-risk labour

    The new clinical practice guidelines included a revised definition of low-risk labour. We, therefore, revised the definition of women having pregnancies with no special high-risk factors or complications (table 1).25 26 34 35 In particular, we did not include ‘mother treated for infertility’ as a risk factor, because infertility treatment does not necessarily result in high-risk labour. However, we did include ‘IVF’ and ‘after extensive infertility treatment’ as high-risk factors because of the potential risk of complications associated. Panel members confirmed the definition used in clinical settings and were asked to approve the new definition at the panel meeting.

    View this table:
    Table 1

    Categories for high-risk factors*

    In Japan, most births are managed in hospitals (530 172, 54.3%) or private obstetric clinics (439 371, 45.0%). Very few take place in a midwifery-led unit (5968, 0.6%) or at home (1168, 0.1%).23 In this study, low-risk labour care was, therefore, limited to care provided by a midwife with obstetrician approval in the late stages of pregnancy, and low-risk labour was defined as labour suitable for in-hospital midwifery with obstetrician approval in the late stages of pregnancy. Low-risk labour is expected to result in normal childbirth and excludes the high-risk factors listed above. Items related to abnormalities during labour or after delivery are excluded

    Step 2: identification of new candidate indicators

    To identify candidate indicators, we extracted existing recommendations from clinical practice guidelines relating to care for low-risk labour, and not limited to those published in Japan. Between November 2015 and January 2016, we searched for data sources using the terms ‘pregnant women’, ‘mothers’, ‘infant’, ‘perinatal care’, ‘prenatal care’, ‘postnatal care’, ‘delivery’, ‘obstetric’ and ‘surgical procedure’. If the databases allowed searches by category, these were used. We searched five databases for guidelines, the US Agency for Healthcare Research and Quality (AHRQ) National Guideline Clearinghouse, Guidelines International Network, Minds from the Japan Council for Quality Health Care, UK National Institute for Health and Care Excellence (NICE) and WHO Guidelines. Two databases, the AHRQ National Quality Measures Clearinghouse and NICE quality standards, were also searched for quality indicators. Several of these databases have inclusion criteria on development procedures, including AHRQ National Guideline Clearinghouse,36 AHRQ National Quality Measures Clearinghouse,37 WHO Handbook38 and Minds.39 All clinical practice guidelines and quality indicators obtained using these databases were therefore expected to fulfil certain methodological criteria.

    We screened the literature to identify quality indicators and clinical practice guidelines published in English or Japanese between June 2012 and December 2015. We excluded any for which the title or abstract included the keywords ‘16 years old and younger’, ‘40 years old and older’, ‘premature delivery’, ‘multiple births’, ‘breech presentation’, ‘prepregnancy obesity’, ‘pregnancy complication’, ‘obstetric history’, ‘abnormal pregnancy progress’, ‘infant congenital disease’, ‘diagnosis and treatment for infant disease’, ‘birth weight under 2000 g or over 4000 g’, ‘anaesthesia’, ‘operation and examination procedure’, ‘28 or less weeks gestation’, ‘more than 1-month post partum’, ‘medical care provided only for high-risk labour’ or ‘does not apply in Japanese clinical practice’. We selected for recommendations on clinical practice by searching for: ‘recommend’, ‘do not recommend’ and ‘suggest’. We did not include ‘weak recommendation or suggestion’, in line with the Grading Recommendations, Assessment, Development and Evaluations method.40 41 The other exclusion criteria were ‘content similar to previous candidate quality indicator’, ‘already basic standard in Japanese clinical practice’, ‘unmeasurable in Japanese clinical systems’ or ‘medical care provided only for high-risk labour’. When we found a revised clinical practice guideline which had been cited in the original development process, we confirmed which recommendations were new or updated, and considered whether the definition of the quality indicators, including the numerator and denominator, should be changed. We also considered the feasibility of data collection and the measurability of data.

    Two researchers both extracted data independently. One assembled the candidate indicators and updated information about the original indicator set. This work was supervised by two senior researchers: a qualitative research specialist and an epidemiologist and guideline methodologist.

    Step 3: convening the multidisciplinary panel

    We aimed to construct a multidisciplinary panel of clinicians, public health specialists and lay mothers from across Japan. We selected obstetricians, paediatricians and midwives with5 or more years of clinical experience in supporting labour or neonatal care. We chose at least two panel members from each specialty to reduce bias in individual opinion. We recruited the participants in the previous panel in 2012, and added other candidates recommended by the original members. We obtained written informed consent from all panel members, and finally, convened a multidisciplinary panel of obstetricians, paediatricians, midwives, public health specialists and mothers who were non-clinicians.

    Step 4: panel ratings (rounds 1–2)

    During each round, panel members rated the appropriateness of each candidate indicator on a nine-point Likert-type scale ranging from 1 ‘least suitable’ to 9 ‘most suitable’. The panel also provided comments on the new indicators and the original 23 and suggested additional candidates. Based on criteria from the US National Quality Forum Measure Evaluation Criteria42 and the American College of Cardiology/American Heart Association,43 panel members rated the appropriateness of new quality indicators by: (1) usefulness in improving outcomes among mothers and infants; (2) whether the measure is clinically relevant; (3) validity; (4) reliability; (5) feasibility of measure implementation and (6) the overall assessment of the indicator (see online supplementary file 1).

    Supplemental material

    In round 2, panel members were shown the anonymous results from round 1 and asked to evaluate any additional candidate indicators and those new indicators on which they had not reached an agreement in round 1. The candidate indicators and their descriptions were modified after discussion with panel members by email or telephone. We provided one or two reminders to any non-respondents. The process was completed in July 2016. All panel members approved the final list of updated indicators.

    Adoption criteria

    Candidate indicators were adopted if the median rating for ‘overall assessment’ during round 1 or round 2 was greater than 7, and if 3 or fewer of the 16 panel members rated it as less than 3.44

    Patient involvement (involvement of mother)

    The lay members of the panel were mothers. They all had to: (1) have given birth, (2) be interested in low-risk labour and midwife-led care and (3) not be clinicians. We first invited the mothers who had participated in the development of the original quality indicators24 and added additional panel members recommended by the original members. Before asking them to agree to participate, we provided information about the aim, significance and methods of the study, the expected work and schedule of participation, and the advantages and burden of participating. After the mothers agreed to participate, we provided them with more information, for example, about quality in medicine, how to rate the quality of healthcare, high quality in healthcare, quality indicators and clinical practice guideline. The mothers participated fully as multidisciplinary panel members, on the same terms as the clinicians. They were also asked to propose new candidate indicators. They received no formal training in clinical guidelines and quality indicators, but we provided them with additional explanations and answered any questions face to face or by email or telephone both before and during the panel.

    Results

    The multidisciplinary panel included 16 people, 13 clinicians (two paediatricians, four obstetricians, five midwives and two epidemiologists) and 3 mothers who were non-clinicians (table 2). All 16 members agreed to participate and responded to the postal questionnaires.

    View this table:
    Table 2

    The composition of the multidisciplinary panel

    The literature review identified 27 clinical practice guidelines, with 277 recommendations, and 21 existing quality indicators (figure 1), which allowed us to develop 11 candidate indicators. We also considered the previous set of 23 indicators for low-risk labour care, all of which were included in the process in their existing form. This provided a set of 34 indicators for low-risk labour care for round 1 of the Delphi process. The panel adopted 10 of the 11 candidate indicators, excluding Indicator 13 (table 3). Two additional candidates (indicators 11 and 12, table 3) were proposed and adopted in round 2 (figure 2). All 23 of the original indicators were readopted (see online supplementary file 2), giving a final set of 35 quality indicators.

    Supplemental material

    View this table:
    Table 3

    The list of new candidate indicators for low-risk labour care

    Figure 1
    Figure 1

    Overview of the literature review process: review of the guidelines and quality indicators extracted to generate candidate indicators.

    Figure 2
    Figure 2

    Process used to update quality indicators: a modified Delphi method for low-risk labour care.

    Discussion

    Using criteria proposed for updating systematic reviews,4 we decided that the quality indicators for low-risk labour care in Japan needed renewal. We updated them using the modified Delphi method, drawing on new or revised clinical practice guidelines and quality indicators, resulting in a set of 35 quality indicators (the original 23 plus 12 new). To our knowledge, this is the first article to describe a systematic and effective way to update quality indicators.

    As new guidelines are developed or revised, it is important to consider whether quality indicators should also be updated. Few previous studies, however, have reported on this process.14–16 18 19 In this study, we refined search strategies and selection criteria used for candidate indicators. First, we added ‘not applicable for Japanese clinical practice’ as an exclusion criterion. This allowed us to exclude, for example, some recommendations from WHO guidelines45 that are primarily designed for use in resource-limited situations in low/middle-income countries. Second, in step 2, we were able to search efficiently by using almost the same strategies and databases as the original research. Third, there was no need to change the original 23 quality indicators once their validity had been checked. Fourth, we kept any unchanged recommendations in revised clinical practice guidelines that had been included in the indicator development work. The work to identify candidates was, therefore, easier than for the original quality indicator development. This strategy could, therefore, be as specific and efficient as updating systematic reviews.4

    In selecting candidate indicators, we recognised that some widely used quality indicators for childbirth might not be applicable to low-risk labour. For example, ‘women with non-medically indicated vaginal deliveries (induction labour) or non-medically indicated caesarean sections at greater than or equal to 37 and less than 39 weeks of gestation completed’46 is widely used as an indicator for childbirth care. However, we did not adopt this because pregnant women may have planned normal vaginal delivery even with a clinical judgement of medical indications. They do not undergo caesarean section or induction labour before 39 weeks 0 day, unless an earlier delivery is necessary because of maternal or fetal indications. A pregnant woman planning normal vaginal delivery is diagnosed by an obstetrician as fit for low-risk vaginal delivery. If she shows signs of complications during her pregnancy, she will be treated immediately. Careful consideration is, therefore, needed of all potential quality indicators.

    The panel composition might have affected the consensus process of developing and updating quality indicators. Few details, however, are available from previous studies.6 This study, therefore, clarified the process used to recruit panel members, including defining the required composition and recruiting accordingly. As a result, of the 16 present members, nine had also participated in the original development. The original panel was also multidisciplinary and included various perinatal care providers and mothers. To make the panel construction process more efficient, we first asked the original panel members to participate, because they had a strong interest in improving quality in low-risk labour and were familiar with the whole process. In addition, we included seven new members to bring new experience and opinions. This enabled the panel to discuss care quality thoroughly from different viewpoints. Patients’ experience will necessarily be individual. When recruiting mothers to participate in the study, we discussed the potential bias of panel members belonging to associations supporting mothers. These associations may have preconceptual views on labour and therefore might influence the rating process through their members. However, these women can also draw on the experience of other mothers, which might improve the rating process. To avoid any imbalance, we ensured that at least one member had no relationship with any such association.

    Incorporating patient views in discussions about quality and safety is considered important, but patient participation in development of quality indicators has been limited.6 47–49 In this study, two new indicators (‘Women having a review of their childbirth experience and support with the midwives and other staff who assisted at the birth’ and ‘Women having been encouraged and supported to adopt the most comfortable positions throughout second stage labour’) were proposed by the mothers on the panel and were eventually adopted as new indicators. They were advanced indicators, designed to evaluate the mothers’ decision-making process. A wide use of these quality indicators, including patients’ views, could play a role in improving the quality of care for both mothers and healthcare providers.

    Limitations

    This study had some limitations. First, not all clinical practice guidelines are quality assured, for example, using the ‘Appraisal of Guidelines and Research and Evaluation’ (AGREEII) instrument.8 However, the databases that were mainly used in this study apply their own quality criteria before they register any clinical practice guidelines.36–38 Second, we did not have the funds or time to organise a face-to-face discussion and needed to add this list of indicators to the next test process. A face-to-face meeting means that all members can offer views, but it has the disadvantage that one member can influence others and affect the ranking. We, therefore, chose the modified Delphi method, rather than the RAND-modified Delphi method, which involves face-to-face meetings. Third, before the Delphi process, we did not have any process or sessions for the three mothers who were panel members to get to know each another and share views. Instead, we provided each of them with information on an individual basis, plus an opportunity to ask questions. However, it is possible that they might have felt more comfortable with the Delphi process, and made more contributions, if we had provided a session for them together in advance. Few quality indicator development processes have involved patients, and even those that have done so have not provided full descriptions of the patient involvement. The strength of this study is that laywomen were included, and a detailed description has been provided of the process. Fourth, despite our intention of developing a method that could be replicable in other clinical fields, there are likely to be topic-specific considerations. The basic conditions for its application would be the existence of solid and sufficient clinical practice guidelines, and guarantee of their quality. With this in place, however, we believe that we have put forward a suitable procedure for how and when to update quality indicators.

    Conclusions

    Through a transparent process, we have updated the set of quality indicators for low-risk labour care in Japan, using a modified Delphi method, and drawing on new clinical practice guidelines and quality indicators. As a next step, we are testing the indicators and will revise the list to improve monitoring and help ensure the provision of safe, high-quality maternity and labour care.

    Acknowledgments

    We are grateful to Hiroyuki Kitajima, Isao Kuwahara, Miho Egawa, Fumitomo Nishimura, Masato Takeuchi, Toshiyuki Sado, Yoshimi Miyaki, Kazue Watanabe, Sachie Nishi, Tomoko Terai, Sachi Nakagami, Makiko Suga, Keiko Miyamoto, Yoko Chiba, Chie Kouta and Masako Senda for their valuable insight and comments. All of them permit us to mention their name. We thank Melissa Leffler from Edanz Group (www.edanzediting.com/ac) for editing a draft of this manuscript.

    References

    1. 1.↵
      Institute of Medicine Committee. Institute of medicine committee to design a strategy for quality, review assurance in medicare. Washington, DC: National Academies Press, 1990.
    2. 2.↵
      2. Boyce MB ,
      3. Browne JP ,
      4. Greenhalgh J
      . The experiences of professionals with using information from patient-reported outcome measures to improve the quality of healthcare: a systematic review of qualitative research. BMJ Qual Saf 2014;23:508–18.doi:10.1136/bmjqs-2013-002524
      OpenUrlAbstract/FREE Full Text
    3. 3.↵
      2. Higashi T ,
      3. Shekelle PG ,
      4. Adams JL , et al
      . Quality of care is associated with survival in vulnerable older patients. Ann Intern Med 2005;143:274–81.doi:10.7326/0003-4819-143-4-200508160-00008
      OpenUrlCrossRefPubMedWeb of Science
    4. 4.↵
      2. Garner P ,
      3. Hopewell S ,
      4. Chandler J , et al
      . When and how to update systematic reviews: consensus and checklist. BMJ 2016;354:i3507.doi:10.1136/bmj.i3507
      OpenUrlFREE Full Text
    5. 5.↵
      2. Bonfill X ,
      3. Roqué M ,
      4. Aller MB , et al
      . Development of quality of care indicators from systematic reviews: the case of hospital delivery. Implement Sci 2013;8:42.doi:10.1186/1748-5908-8-42
      OpenUrlCrossRefPubMed
    6. 6.↵
      2. Kötter T ,
      3. Blozik E ,
      4. Scherer M
      . Methods for the guideline-based development of quality indicators--a systematic review. Implement Sci 2012;7:21.doi:10.1186/1748-5908-7-21
      OpenUrlCrossRefPubMed
    7. 7.↵
      2. Shekelle PG ,
      3. Ortiz E ,
      4. Rhodes S , et al
      . Validity of the agency for healthcare research and quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA 2001;286:1461–7.
      OpenUrlCrossRefPubMedWeb of Science
    8. 8.↵
      2. Brouwers MC ,
      3. Kho ME ,
      4. Browman GP , et al
      . AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010;182:E839–42.doi:10.1503/cmaj.090449
      OpenUrlFREE Full Text
    9. 9.↵
      2. Venkatesh AK ,
      3. Savage D ,
      4. Sandefur B , et al
      . Systematic review of emergency medicine clinical practice guidelines: Implications for research and policy. PLoS One 2017;12:e0178456.doi:10.1371/journal.pone.0178456
    10. 10.↵
      2. Antman EM ,
      3. Lau J ,
      4. Kupelnick B , et al
      . A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA 1992;268:240–8.
      OpenUrlCrossRefPubMedWeb of Science
    11. 11.↵
      2. Wright JD ,
      3. Pawar N ,
      4. Gonzalez JS , et al
      . Scientific evidence underlying the American College of Obstetricians and Gynecologists’ practice bulletins. Obstet Gynecol 2011;118:505–12.doi:10.1097/AOG.0b013e3182267f43
      OpenUrlCrossRefPubMed
    12. 12.↵
      2. Jacobs AK ,
      3. Anderson JL ,
      4. Halperin JL , et al
      . The evolution and future of ACC/AHA clinical practice guidelines: a 30-year journey: a report of the American College of Cardiology/American Heart Association Task Force on practice guidelines. Circulation 2014;130:1208–17.doi:10.1161/CIR.0000000000000090
      OpenUrlFREE Full Text
    13. 13.↵
      2. Saver BG ,
      3. Martin SA ,
      4. Adler RN , et al
      . Care that matters: quality measurement and health care. PLoS Med 2015;12:e1001902.doi:10.1371/journal.pmed.1001902
    14. 14.↵
      2. Thomas RJ ,
      3. King M ,
      4. Lui K , et al
      . AACVPR/ACCF/AHA 2010 Update: Performance Measures on Cardiac Rehabilitation for Referral to Cardiac Rehabilitation/Secondary Prevention Services Endorsed by the American College of Chest Physicians, the American College of Sports Medicine, the American Physical Therapy Association, the Canadian Association of Cardiac Rehabilitation, the Clinical Exercise Physiology Association, the European Association for Cardiovascular Prevention and Rehabilitation, the Inter-American Heart Foundation, the National Association of Clinical Nurse Specialists, the Preventive Cardiovascular Nurses Association, and the Society of Thoracic Surgeons. J Am Coll Cardiol 2010;56:1159–67.doi:10.1016/j.jacc.2010.06.006
      OpenUrlFREE Full Text
    15. 15.↵
      2. Drozda JP ,
      3. Ferguson TB ,
      4. Jneid H , et al
      . 2015 ACC/AHA Focused Update of Secondary Prevention Lipid Performance Measures: A Report of the American College of Cardiology/American Heart Association Task Force on Performance Measures. J Am Coll Cardiol 2016;67:558–87.doi:10.1016/j.jacc.2015.02.003
      OpenUrlFREE Full Text
    16. 16.↵
      2. Sanders AE ,
      3. Nininger J ,
      4. Absher J , et al
      . Quality improvement in neurology: Dementia management quality measurement set update. Neurology 2017;88:1951–7.doi:10.1212/WNL.0000000000003917
      OpenUrl
    17. 17.↵
      2. Weled BJ ,
      3. Adzhigirey LA ,
      4. Hodgman TM , et al
      . Critical care delivery: the importance of process of care and icu structure to improved outcomes: an update from the american college of critical care medicine task force on models of critical care. Crit Care Med 2015;43:1520–5.doi:10.1097/CCM.0000000000000978
      OpenUrlCrossRefPubMed
    18. 18.↵
      2. Wiler JL ,
      3. Welch S ,
      4. Pines J , et al
      . Emergency department performance measures updates: proceedings of the 2014 emergency department benchmarking alliance consensus summit. Acad Emerg Med 2015;22:542–53.doi:10.1111/acem.12654
      OpenUrl
    19. 19.↵
      2. Carinci F ,
      3. Van Gool K ,
      4. Mainz J , et al
      . Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care 2015;27:137–46.doi:10.1093/intqhc/mzv004
      OpenUrlCrossRefPubMed
    20. 20.↵
      2. Kleefstra SM ,
      3. Zandbelt LC ,
      4. de Haes HJ , et al
      . Trends in patient satisfaction in Dutch university medical centers: room for improvement for all. BMC Health Serv Res 2015;15:112.doi:10.1186/s12913-015-0766-7
      OpenUrlCrossRefPubMed
    21. 21.↵
      2. Grol RBR ,
      3. Moss F
      . Quality improvement research: understanding the science of change in health care. BMJ Publish Group 2004.
    22. 22.↵
      Ministry of Health, Labour and Welfare, Japan. Vital statistics of Japan 2015: National Statistics Center, Statistics Japan, 2016.
    23. 23.↵
      Japanese Nurse Association. Chapter 5 Risk scores and medical institutions for parturient. In: Survey on temporary relocation system of midwife and acceptability of midwifery practice etc in 2012, 2014.
    24. 24.↵
      2. Ueda K ,
      3. Ohtera S ,
      4. Kaso M , et al
      . Development of quality indicators for low-risk labor care provided by midwives using a RAND-modified Delphi method. BMC Pregnancy Childbirth 2017;17:315.doi:10.1186/s12884-017-1468-4
      OpenUrl
    25. 25.↵
      Japan Society of Obstetrics and Gynecology and Japan Association of Obstetricians and Gynecologists. Guidelines for obstetrical practice in Japan. 2014. http://www.jsog.or.jp/activity/pdf/gl_sanka_2014.pdf (Accessed 21 Jan 2018).
    26. 26.↵
      Japanese Midwives Assosiation. Midwife Clinical Practice Guidelines. 2014. http://www.midwife.or.jp/pdf/guideline/guideline.pdf (Accessed 21 Jan 2018).
    27. 27.↵
      General Incorporated Assosiation Japan Acadeny of Midwifery. 2012 Evidence-based guidelines for midwifery care during childbirth. 2012. http://square.umin.ac.jp/~jam/docs/ebm_guideline_childbirth2012.pdf (Accessed 21 Jan 2018).
    28. 28.↵
      2. Blozik E ,
      3. Nothacker M ,
      4. Bunk T , et al
      . Simultaneous development of guidelines and quality indicators -- how do guideline groups act? A worldwide survey. Int J Health Care Qual Assur 2012;25:712–29.doi:10.1108/09526861211270659
      OpenUrl
    29. 29.↵
      2. Campbell SM ,
      3. Braspenning J ,
      4. Hutchinson A , et al
      . Research methods used in developing and applying quality indicators in primary care. BMJ 2003;326:816–9.doi:10.1136/bmj.326.7393.816
      OpenUrlFREE Full Text
    30. 30.↵
      2. Higashi T
      . Lessons learned in the development of process quality indicators for cancer care in Japan. Biopsychosoc Med 2010;4:14.doi:10.1186/1751-0759-4-14
      OpenUrlCrossRefPubMed
    31. 31.↵
      2. Mukai H ,
      3. Higashi T ,
      4. Sasaki M , et al
      . Quality evaluation of medical care for breast cancer in Japan. Int J Qual Health Care 2016;28:110–3.doi:10.1093/intqhc/mzv109
      OpenUrlCrossRefPubMed
    32. 32.↵
      2. Nothacker M ,
      3. Stokes T ,
      4. Shaw B , et al
      . Reporting standards for guideline-based performance measures. Implement Sci 2016;11:6.doi:10.1186/s13012-015-0369-z
      OpenUrl
    33. 33.↵
      2. Murphy MK ,
      3. Black NA ,
      4. Lamping DL , et al
      . Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998;2:1–88.
      OpenUrlPubMed
    34. 34.↵
      2. Nakabayashi M
      . Study on safety steps in obstetrics department. Health Labour Science Research Grant Research on Health Technology Assessment, The summary and sharing documentation in 2004. 2004:1–16.
    35. 35.↵
      Japan Society of Obstetrics and Gynecology. Social insurance ABC for obstetricians. 2016.
    36. 36.↵
      AHRQ National Guideline Clearinghouse. Inclusion criteria. 2014. https://www.guideline.gov/help-and-about/summaries/inclusion-criteria (Accessed 21 Jan 2018).
    37. 37.↵
      AHRQ National Quality Measures Clearinghouse. Inclusion Criteria. 2017. https://qualitymeasures.ahrq.gov/help-and-about/summaries/inclusion-criteria (Accessed 21 Jan 2018).
    38. 38.↵
      World Health Organization. WHO Handbook for Guideline Development. 2nd edn, 2014.
    39. 39.↵
      2. Brouwers MC ,
      3. Kho ME ,
      4. Browman GP , et al
      . AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol 2010;63:1308–11.doi:10.1016/j.jclinepi.2010.07.001
      OpenUrlCrossRefPubMed
    40. 40.↵
      2. Andrews JC ,
      3. Schünemann HJ ,
      4. Oxman AD , et al
      . GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726–35.doi:10.1016/j.jclinepi.2013.02.003
      OpenUrlCrossRefPubMed
    41. 41.↵
      2. Guyatt GH ,
      3. Oxman AD ,
      4. Vist GE , et al
      . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.doi:10.1136/bmj.39489.470347.AD
      OpenUrlFREE Full Text
    42. 42.↵
      National Quality Forum. Measure evaluation criteria and guidance for evaluating measures for endorsement. 2016. http://www.qualityforum.org/Measuring_Performance/Submitting_Standards/Measure_Evaluation_Criteria.aspx (Accessed 21 Jan 2018).
    43. 43.↵
      2. Spertus JA ,
      3. Eagle KA ,
      4. Krumholz HM , et al
      . American College of Cardiology and American Heart Association methodology for the selection and creation of performance measures for quantifying the quality of cardiovascular care. Circulation 2005;111:1703–12.doi:10.1161/01.CIR.0000157096.95223.D7
      OpenUrlAbstract/FREE Full Text
    44. 44.↵
      2. Fitch K ,
      3. Burnand B ,
      4. LaCalle JR , et al
      . The RAND/UCLA appropriateness method user’s manual : RAND Corporation, 2001.
    45. 45.↵
      World Health Organization. Guidelines on maternal, newborn, child and adolescent health Listed by year of publication. http://www.who.int/maternal_child_adolescent/guidelines/guidelines-year/en/ (Accessed 21 Jan 2018).
    46. 46.↵
      Joint Commission. Perinatal care: percentage of patients with elective vaginal deliveries or elective cesarean births at greater than or equal to 37 and less than 39 weeks of gestation completed: United States Agency for Healthcare Research and Quality (AHRQ) National Guideline Clearinghouse, 2017.
    47. 47.↵
      2. Kerr EA ,
      3. Hayward RA
      . Patient-centered performance management: enhancing value for patients and health care systems. JAMA 2013;310:137–8.doi:10.1001/jama.2013.6828
      OpenUrlCrossRefPubMedWeb of Science
    48. 48.↵
      2. den Breejen EM ,
      3. Nelen WL ,
      4. Schol SF , et al
      . Development of guideline-based indicators for patient-centredness in fertility care: what patients add. Hum Reprod 2013;28:987–96.doi:10.1093/humrep/det010
      OpenUrlCrossRefPubMedWeb of Science
    49. 49.↵
      Medical Information Network Distribution Service. The method of evaluation and selection for clinical practice guideline. 2016. http://minds.jcqhc.or.jp/s/about_us_evaluation_method (Accessed 10 Jan 2018).

    Footnotes

    • Contributors KU, MK, SO and TN planned the study. KU and MK independently extracted evidence and TN supervised the process. KU, MK, SO and TN prepared and carried out the modified Delphi procedure. KU conducted the data analyses and prepared the manuscript. SO and TN critically commented on the manuscript. All authors read and approved the final manuscript.

    • Funding This work was supported by the Ministry of Education, Culture, Sports, Science and Technology, (Grant-in-Aid for Young Scientists B, 26870745, 2014-2017) and Health and Labor Sciences Research Grant.

    • Competing interests None declared.

    • Ethics approval This study was approved by the Ethics Committee of Kyoto University Graduate School and Faculty of Medicine.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.

    • Patient consent for publication Not required.