Article Text

Download PDFPDF

Updating quality indicators for low-risk labour care in Japan using current clinical practice guidelines: a modified Delphi method
  1. Kayo Ueda1,2,
  2. Misato Kaso1,
  3. Shosuke Ohtera1,
  4. Takeo Nakayama1
  1. 1 Department of Health Informatics in the School of Public Health, Kyoto University, Kyoto, Japan
  2. 2 Graduate School of Nursing Women’s Health & Midwifery, School of Medicine Faculty of Nursing Department of Maternal Nursing, Nara Medical University, Kashihara, Japan
  1. Correspondence to Dr Kayo Ueda; ueda.kayo.85a{at}


Objectives Quality indicators are measurable elements widely used to assess the quality of care. They are often developed from the results of systematic reviews or clinical practice guidelines. These sources are regularly updated in line with new clinical evidence, but there are few articles on updating quality indicators based on clinical practice guidelines. This study aimed to update the quality indicators developed for low-risk labour care in Japan in 2012, mainly drawing on new or updated clinical practice guidelines, and making the process clearly visible and assessable.

Design and setting We used a modified Delphi method for the update. The procedure included four steps: (1) updating the definition of low-risk labour; (2) reviewing the literature published between June 2012 and December 2015 using five guidelines and two quality indicator databases to extract potential candidate indicators; (3) formation of a multidisciplinary panel including mothers and (4) panel ratings (two rounds between February and April 2016) on the validity of the candidate indicators, and judging the validity of the previous quality indicators drawing on the new evidence.

Participants A multidisciplinary panel of 13 clinicians, including obstetricians, paediatricians and midwives, plus 3 non-clinician mothers.

Results The literature review identified 276 new recommendations from 27 clinical practice guidelines including 2 published in Japan and 21 quality indicators. We developed 13 new candidate indicators from these sources and panel recommendations, 12 of which were approved by the multidisciplinary panel. The panel also accepted all 23 existing quality indicators as still valid, resulting in a total of 35 quality indicators for low-risk labour.

Conclusions We successfully updated the quality indicators for low-risk labour care in Japan. The procedure developed may be useful for updating other quality indicators based on new clinical practice guidelines.

  • quality indicator
  • clinical practice guidelines
  • update
  • modified delphi method
  • low-risk labor

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • Drawing on current clinical practice guidelines, we efficiently updated the quality indicators for low-risk pregnancy care using a modified Delphi method.

  • After clarifying the method for recruiting a multidisciplinary panel, our panel included 13 clinicians and 3 mothers who were non-clinicians.

  • The panel composition may have affected the consensus process of developing and updating quality indicators.

  • This update procedure may be useful for updating quality indicators based on revised or new clinical practice guidelines in other fields.


The US Institute of Medicine has defined the quality of care as ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’.1 Quality indicators are measurable elements that are widely used to assess the quality of care. They are usually drawn up from up-to-date published evidence including systematic reviews and clinical practice guidelines.2 3

It has long been recognised as necessary to update clinical practice guidelines and systematic reviews as new relevant studies are published.4–7 This suggests that it is also important to update quality indicators based on clinical practice guidelines and systematic reviews,8–13 but few articles have ever reported on this updating process. If clinical practice guidelines make new recommendations, it is essential to discuss whether to incorporate these into existing quality indicators.4

There are four reasons to update quality indicators. The first is because it is necessary to reassess the evidence, for example, because new clinical practice guidelines have been published.14–17 The second is during a review of concepts or definitions of expert terms, such as revision of definitions and metrics.18 19 The third is when adjusting the data collection system or method used by healthcare providers, such as updates to indicator content or clarifying and adjusting a particular procedure.14 19 The fourth is when there is a ceiling effect. We defined the ‘ceiling effect’ as a phenomenon in which the scores for the quality indicator are near their maximum value. If a ceiling effect occurs, it is impossible to increase the score significantly. It is, therefore, also very difficult to evaluate the quality improvement or detect differences between the measured score and other hospitals or over time.20 21 Very few previous reports have sufficiently described the methods and processes used to update quality indicators.6

In Japan, quality indicators have rarely linked to clinical practice guidelines. There is no national data collection or reporting on patient safety in Japan, and this is not monitored nor shared across hospitals. The authors developed quality indicators for low-risk labour care in 2012. The annual number of live births in Japan is about 980 000, of which 92% are normal term infants. In 2016, the maternal mortality rate was 3.4 and the perinatal mortality rate was 3.6.22

The majority involve low-risk labour,23 which seldom results in complications. We used clinical practice guidelines and quality indicators which were related to low-risk birth and published between July 2007 and June 2012. The results were published in English in 2017.24

Quality indicators for low-risk labour care are necessary for several reasons. First, healthcare providers need to share a common understanding of checks that will keep birth-related risks low, reducing potential complications and problems caused by insufficient communication. Second, healthcare providers can compare the quality of care with other facilities, to improve their practice. Third, quality indicators can be used to ensure midwives’ accountability for low-risk labour care.

More recently in Japan, there has been an increasingly uneven distribution of obstetricians, a decrease in facilities specialised for childbirth. It has become difficult for obstetricians to be responsible for all low-risk births in the hospital, so systems allowing midwife-led care are being examined. It is, therefore, even more important to guarantee the quality of care for low-risk labour.

Since the development of the original quality indicators in 2012, new clinical practice guidelines on low-risk labour have been published in Japan, which suggests the need to reconsider the quality indicators.25–27 This study, therefore, aimed to update the quality indicators for low-risk labour care drawing on new clinical practice guidelines and to propose an efficient way to update quality indicators.


Indicator update

Review of the literature to assess whether any updating was necessary

We assessed whether the information in recently published clinical practice guidelines touching on low-risk labour care could affect the original quality indicators. We also made an abbreviated search of previously cited clinical practice guidelines to identify any new recommendations or information. Enough new information emerged that we decided that it was necessary to update the quality indicators for low-risk labour.

The update process

We used a modified Delphi method to integrate scientific evidence with expert panel opinions. This method has been widely used to develop quality indicators.2 6 28–33 To develop a consensus, panel members individually rated candidate indicators in two rounds, through a postal questionnaire, with the second round used to resolve any issues emerging from round 1. In the second round, the rating results and all comments, including agreement and disagreement from the first round, were shared anonymously with the panel members. Disagreements were discussed by email or telephone between a facilitator (a researcher: KU) and the panel member to assure anonymity of comments. We used the consensus-based reporting standards for guideline-based performance measure development and re-evaluation proposed by the Guidelines International Network Performance Measures Working Group.32

Step 1: update of the definition of low-risk labour

The new clinical practice guidelines included a revised definition of low-risk labour. We, therefore, revised the definition of women having pregnancies with no special high-risk factors or complications (table 1).25 26 34 35 In particular, we did not include ‘mother treated for infertility’ as a risk factor, because infertility treatment does not necessarily result in high-risk labour. However, we did include ‘IVF’ and ‘after extensive infertility treatment’ as high-risk factors because of the potential risk of complications associated. Panel members confirmed the definition used in clinical settings and were asked to approve the new definition at the panel meeting.

Table 1

Categories for high-risk factors*

In Japan, most births are managed in hospitals (530 172, 54.3%) or private obstetric clinics (439 371, 45.0%). Very few take place in a midwifery-led unit (5968, 0.6%) or at home (1168, 0.1%).23 In this study, low-risk labour care was, therefore, limited to care provided by a midwife with obstetrician approval in the late stages of pregnancy, and low-risk labour was defined as labour suitable for in-hospital midwifery with obstetrician approval in the late stages of pregnancy. Low-risk labour is expected to result in normal childbirth and excludes the high-risk factors listed above. Items related to abnormalities during labour or after delivery are excluded

Step 2: identification of new candidate indicators

To identify candidate indicators, we extracted existing recommendations from clinical practice guidelines relating to care for low-risk labour, and not limited to those published in Japan. Between November 2015 and January 2016, we searched for data sources using the terms ‘pregnant women’, ‘mothers’, ‘infant’, ‘perinatal care’, ‘prenatal care’, ‘postnatal care’, ‘delivery’, ‘obstetric’ and ‘surgical procedure’. If the databases allowed searches by category, these were used. We searched five databases for guidelines, the US Agency for Healthcare Research and Quality (AHRQ) National Guideline Clearinghouse, Guidelines International Network, Minds from the Japan Council for Quality Health Care, UK National Institute for Health and Care Excellence (NICE) and WHO Guidelines. Two databases, the AHRQ National Quality Measures Clearinghouse and NICE quality standards, were also searched for quality indicators. Several of these databases have inclusion criteria on development procedures, including AHRQ National Guideline Clearinghouse,36 AHRQ National Quality Measures Clearinghouse,37 WHO Handbook38 and Minds.39 All clinical practice guidelines and quality indicators obtained using these databases were therefore expected to fulfil certain methodological criteria.

We screened the literature to identify quality indicators and clinical practice guidelines published in English or Japanese between June 2012 and December 2015. We excluded any for which the title or abstract included the keywords ‘16 years old and younger’, ‘40 years old and older’, ‘premature delivery’, ‘multiple births’, ‘breech presentation’, ‘prepregnancy obesity’, ‘pregnancy complication’, ‘obstetric history’, ‘abnormal pregnancy progress’, ‘infant congenital disease’, ‘diagnosis and treatment for infant disease’, ‘birth weight under 2000 g or over 4000 g’, ‘anaesthesia’, ‘operation and examination procedure’, ‘28 or less weeks gestation’, ‘more than 1-month post partum’, ‘medical care provided only for high-risk labour’ or ‘does not apply in Japanese clinical practice’. We selected for recommendations on clinical practice by searching for: ‘recommend’, ‘do not recommend’ and ‘suggest’. We did not include ‘weak recommendation or suggestion’, in line with the Grading Recommendations, Assessment, Development and Evaluations method.40 41 The other exclusion criteria were ‘content similar to previous candidate quality indicator’, ‘already basic standard in Japanese clinical practice’, ‘unmeasurable in Japanese clinical systems’ or ‘medical care provided only for high-risk labour’. When we found a revised clinical practice guideline which had been cited in the original development process, we confirmed which recommendations were new or updated, and considered whether the definition of the quality indicators, including the numerator and denominator, should be changed. We also considered the feasibility of data collection and the measurability of data.

Two researchers both extracted data independently. One assembled the candidate indicators and updated information about the original indicator set. This work was supervised by two senior researchers: a qualitative research specialist and an epidemiologist and guideline methodologist.

Step 3: convening the multidisciplinary panel

We aimed to construct a multidisciplinary panel of clinicians, public health specialists and lay mothers from across Japan. We selected obstetricians, paediatricians and midwives with5 or more years of clinical experience in supporting labour or neonatal care. We chose at least two panel members from each specialty to reduce bias in individual opinion. We recruited the participants in the previous panel in 2012, and added other candidates recommended by the original members. We obtained written informed consent from all panel members, and finally, convened a multidisciplinary panel of obstetricians, paediatricians, midwives, public health specialists and mothers who were non-clinicians.

Step 4: panel ratings (rounds 1–2)

During each round, panel members rated the appropriateness of each candidate indicator on a nine-point Likert-type scale ranging from 1 ‘least suitable’ to 9 ‘most suitable’. The panel also provided comments on the new indicators and the original 23 and suggested additional candidates. Based on criteria from the US National Quality Forum Measure Evaluation Criteria42 and the American College of Cardiology/American Heart Association,43 panel members rated the appropriateness of new quality indicators by: (1) usefulness in improving outcomes among mothers and infants; (2) whether the measure is clinically relevant; (3) validity; (4) reliability; (5) feasibility of measure implementation and (6) the overall assessment of the indicator (see online supplementary file 1).

Supplemental material

In round 2, panel members were shown the anonymous results from round 1 and asked to evaluate any additional candidate indicators and those new indicators on which they had not reached an agreement in round 1. The candidate indicators and their descriptions were modified after discussion with panel members by email or telephone. We provided one or two reminders to any non-respondents. The process was completed in July 2016. All panel members approved the final list of updated indicators.

Adoption criteria

Candidate indicators were adopted if the median rating for ‘overall assessment’ during round 1 or round 2 was greater than 7, and if 3 or fewer of the 16 panel members rated it as less than 3.44

Patient involvement (involvement of mother)

The lay members of the panel were mothers. They all had to: (1) have given birth, (2) be interested in low-risk labour and midwife-led care and (3) not be clinicians. We first invited the mothers who had participated in the development of the original quality indicators24 and added additional panel members recommended by the original members. Before asking them to agree to participate, we provided information about the aim, significance and methods of the study, the expected work and schedule of participation, and the advantages and burden of participating. After the mothers agreed to participate, we provided them with more information, for example, about quality in medicine, how to rate the quality of healthcare, high quality in healthcare, quality indicators and clinical practice guideline. The mothers participated fully as multidisciplinary panel members, on the same terms as the clinicians. They were also asked to propose new candidate indicators. They received no formal training in clinical guidelines and quality indicators, but we provided them with additional explanations and answered any questions face to face or by email or telephone both before and during the panel.


The multidisciplinary panel included 16 people, 13 clinicians (two paediatricians, four obstetricians, five midwives and two epidemiologists) and 3 mothers who were non-clinicians (table 2). All 16 members agreed to participate and responded to the postal questionnaires.

Table 2

The composition of the multidisciplinary panel

The literature review identified 27 clinical practice guidelines, with 277 recommendations, and 21 existing quality indicators (figure 1), which allowed us to develop 11 candidate indicators. We also considered the previous set of 23 indicators for low-risk labour care, all of which were included in the process in their existing form. This provided a set of 34 indicators for low-risk labour care for round 1 of the Delphi process. The panel adopted 10 of the 11 candidate indicators, excluding Indicator 13 (table 3). Two additional candidates (indicators 11 and 12, table 3) were proposed and adopted in round 2 (figure 2). All 23 of the original indicators were readopted (see online supplementary file 2), giving a final set of 35 quality indicators.

Supplemental material

Table 3

The list of new candidate indicators for low-risk labour care

Figure 1

Overview of the literature review process: review of the guidelines and quality indicators extracted to generate candidate indicators.

Figure 2

Process used to update quality indicators: a modified Delphi method for low-risk labour care.


Using criteria proposed for updating systematic reviews,4 we decided that the quality indicators for low-risk labour care in Japan needed renewal. We updated them using the modified Delphi method, drawing on new or revised clinical practice guidelines and quality indicators, resulting in a set of 35 quality indicators (the original 23 plus 12 new). To our knowledge, this is the first article to describe a systematic and effective way to update quality indicators.

As new guidelines are developed or revised, it is important to consider whether quality indicators should also be updated. Few previous studies, however, have reported on this process.14–16 18 19 In this study, we refined search strategies and selection criteria used for candidate indicators. First, we added ‘not applicable for Japanese clinical practice’ as an exclusion criterion. This allowed us to exclude, for example, some recommendations from WHO guidelines45 that are primarily designed for use in resource-limited situations in low/middle-income countries. Second, in step 2, we were able to search efficiently by using almost the same strategies and databases as the original research. Third, there was no need to change the original 23 quality indicators once their validity had been checked. Fourth, we kept any unchanged recommendations in revised clinical practice guidelines that had been included in the indicator development work. The work to identify candidates was, therefore, easier than for the original quality indicator development. This strategy could, therefore, be as specific and efficient as updating systematic reviews.4

In selecting candidate indicators, we recognised that some widely used quality indicators for childbirth might not be applicable to low-risk labour. For example, ‘women with non-medically indicated vaginal deliveries (induction labour) or non-medically indicated caesarean sections at greater than or equal to 37 and less than 39 weeks of gestation completed’46 is widely used as an indicator for childbirth care. However, we did not adopt this because pregnant women may have planned normal vaginal delivery even with a clinical judgement of medical indications. They do not undergo caesarean section or induction labour before 39 weeks 0 day, unless an earlier delivery is necessary because of maternal or fetal indications. A pregnant woman planning normal vaginal delivery is diagnosed by an obstetrician as fit for low-risk vaginal delivery. If she shows signs of complications during her pregnancy, she will be treated immediately. Careful consideration is, therefore, needed of all potential quality indicators.

The panel composition might have affected the consensus process of developing and updating quality indicators. Few details, however, are available from previous studies.6 This study, therefore, clarified the process used to recruit panel members, including defining the required composition and recruiting accordingly. As a result, of the 16 present members, nine had also participated in the original development. The original panel was also multidisciplinary and included various perinatal care providers and mothers. To make the panel construction process more efficient, we first asked the original panel members to participate, because they had a strong interest in improving quality in low-risk labour and were familiar with the whole process. In addition, we included seven new members to bring new experience and opinions. This enabled the panel to discuss care quality thoroughly from different viewpoints. Patients’ experience will necessarily be individual. When recruiting mothers to participate in the study, we discussed the potential bias of panel members belonging to associations supporting mothers. These associations may have preconceptual views on labour and therefore might influence the rating process through their members. However, these women can also draw on the experience of other mothers, which might improve the rating process. To avoid any imbalance, we ensured that at least one member had no relationship with any such association.

Incorporating patient views in discussions about quality and safety is considered important, but patient participation in development of quality indicators has been limited.6 47–49 In this study, two new indicators (‘Women having a review of their childbirth experience and support with the midwives and other staff who assisted at the birth’ and ‘Women having been encouraged and supported to adopt the most comfortable positions throughout second stage labour’) were proposed by the mothers on the panel and were eventually adopted as new indicators. They were advanced indicators, designed to evaluate the mothers’ decision-making process. A wide use of these quality indicators, including patients’ views, could play a role in improving the quality of care for both mothers and healthcare providers.


This study had some limitations. First, not all clinical practice guidelines are quality assured, for example, using the ‘Appraisal of Guidelines and Research and Evaluation’ (AGREEII) instrument.8 However, the databases that were mainly used in this study apply their own quality criteria before they register any clinical practice guidelines.36–38 Second, we did not have the funds or time to organise a face-to-face discussion and needed to add this list of indicators to the next test process. A face-to-face meeting means that all members can offer views, but it has the disadvantage that one member can influence others and affect the ranking. We, therefore, chose the modified Delphi method, rather than the RAND-modified Delphi method, which involves face-to-face meetings. Third, before the Delphi process, we did not have any process or sessions for the three mothers who were panel members to get to know each another and share views. Instead, we provided each of them with information on an individual basis, plus an opportunity to ask questions. However, it is possible that they might have felt more comfortable with the Delphi process, and made more contributions, if we had provided a session for them together in advance. Few quality indicator development processes have involved patients, and even those that have done so have not provided full descriptions of the patient involvement. The strength of this study is that laywomen were included, and a detailed description has been provided of the process. Fourth, despite our intention of developing a method that could be replicable in other clinical fields, there are likely to be topic-specific considerations. The basic conditions for its application would be the existence of solid and sufficient clinical practice guidelines, and guarantee of their quality. With this in place, however, we believe that we have put forward a suitable procedure for how and when to update quality indicators.


Through a transparent process, we have updated the set of quality indicators for low-risk labour care in Japan, using a modified Delphi method, and drawing on new clinical practice guidelines and quality indicators. As a next step, we are testing the indicators and will revise the list to improve monitoring and help ensure the provision of safe, high-quality maternity and labour care.


We are grateful to Hiroyuki Kitajima, Isao Kuwahara, Miho Egawa, Fumitomo Nishimura, Masato Takeuchi, Toshiyuki Sado, Yoshimi Miyaki, Kazue Watanabe, Sachie Nishi, Tomoko Terai, Sachi Nakagami, Makiko Suga, Keiko Miyamoto, Yoko Chiba, Chie Kouta and Masako Senda for their valuable insight and comments. All of them permit us to mention their name. We thank Melissa Leffler from Edanz Group ( for editing a draft of this manuscript.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.


  • Contributors KU, MK, SO and TN planned the study. KU and MK independently extracted evidence and TN supervised the process. KU, MK, SO and TN prepared and carried out the modified Delphi procedure. KU conducted the data analyses and prepared the manuscript. SO and TN critically commented on the manuscript. All authors read and approved the final manuscript.

  • Funding This work was supported by the Ministry of Education, Culture, Sports, Science and Technology, (Grant-in-Aid for Young Scientists B, 26870745, 2014-2017) and Health and Labor Sciences Research Grant.

  • Competing interests None declared.

  • Ethics approval This study was approved by the Ethics Committee of Kyoto University Graduate School and Faculty of Medicine.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available.

  • Patient consent for publication Not required.