Article Text
Abstract
Objectives The maternal near-miss case review (NMCR) cycle is a type of clinical audit aiming at improving quality of maternal healthcare by discussing near-miss cases. In several countries this approach has been introduced and supported by WHO and partners since 2004, but information on the quality of its implementation is missing. This study aimed at evaluating the quality of the NMCR implementation in selected countries within WHO European Region.
Design Cross-sectional study.
Settings Twenty-three maternity units in Armenia, Georgia, Latvia, Moldova and Uzbekistan.
Assessment tools A predefined checklist including 50 items, according to WHO methodology. Quality in the NMCR implementation was defined by summary scores ranging from 0 (totally inappropriate) to 3 (appropriate).
Results Quality of the NMCR implementation was heterogeneous among different countries, and within the same country. Overall, the first part of the audit cycle (from case identification to case analysis) was fairly well performed (mean score 2.00, 95% CI 1.94 to 2.06), with the exception of the ‘inclusion of users’ views’ (mean score 0.66, 95% CI 0.11 to 1.22), while the second part (developing recommendations, implementing them and ensuring quality) was poorly performed (mean score 0.66, 95% CI 0.11 to 1.22). Each country had at least one champion facility, where quality of the NMCR cycle was acceptable. Quality of the implementation was not associated with its duration. Gaps in implementation were of technical, organisational and attitudinal nature.
Conclusions Ensuring quality in the NMCR may be difficult but achievable. The high heterogeneity in results within the same country suggests that quality of the NMCR implementation depends, to a large extent, from hospital factors, including staff’s commitment, managerial support and local coordination. Efforts should be put in preventing and mitigating common barriers that hamper successful NMCR implementation.
- maternal health
- near-miss case review
- standard-based assessment
- quality of care
- middle-income countries
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first study reporting on the quality of the hospital-based near-miss case review (NMCR) in Central Asia and Eastern Europe.
The assessment included five countries within WHO European Region and was based on a predefined checklist, providing the opportunity to evaluate the implementation of the NMCR approach in a standardised manner.
In three countries facilities included in the evaluation accounted for all facilities implementing the NMCR within in the country. In the remaining two countries, where the NMCR was implemented in more hospitals, facilities were chosen in dialogue with local authorities (non-probability sampling), and not at random; however, criteria used to select facilities included also geographical distribution (ie, so that different regions were represented) and hospital type (ie, different types of hospitals were selected).
Introduction
Ensuring adequate quality of healthcare is a primary objective of WHO Global Strategy for Women’s, Children’s and Adolescent’s Health 2016–20301 and of Health 2020, the European strategic framework setting the policy directions for the 53 member states in WHO European Region.2 Quality in healthcare is recognised as essential for the health and well-being of the population, and as a basic aspect of human rights.3–5
Among the different strategies aiming at improving quality of care at maternity services, the facility-based maternal near-miss case review (NMCR) cycle was proposed by WHO in 2004 as a type of clinical audit.6–8 With respect to mortality audit, the NMCR has the advantage to imply less legal issues, and is therefore perceived as more acceptable by staff. Near-miss cases are defined as a woman who nearly died but survived a complication that occurred during pregnancy, childbirth or within 6 weeks after pregnancy.9 In the facility-based NMCR all hospital staff involved in the management of the chosen near-miss case—including obstetricians, midwives, nurses and ancillary staff—get together to discuss and evaluate the care provided against national evidence-based guidelines, local protocols and standards of care. The aim of the case review is to critically discuss local management, procedures and attitudes, and to identify areas that can be further improved.9 Actions to improve quality of maternal healthcare are proposed and agreed by hospital staff, and subsequently monitored to check their implementation, as for a continuous quality improvement process.9 One of the key characteristics of this method is the bottom-up approach, aiming at facilitating local ownership of the process, commitment in implementing the proposed recommendations and team building. Currently, the review of severe maternal morbidity cases (‘near-miss’ events) is recommended by WHO as a key action to eliminate avoidable maternal and perinatal mortality and morbidity and improve the quality of care.10
While in some countries within WHO European Region (such as UK, Norway and the Netherlands) the practice of reviewing maternal near-miss cases was introduced by the government or by professional associations, in several other countries (most often middle-income countries) its implementation was assisted by WHO and/or United Nations Population Fund (UNFPA). In the latter scenario, coverage and quality of the NMCR implementation were usually discussed during workshops,11–13 but so far they have not been evaluated using a systematic methodology.
In 2015, WHO developed a checklist for assessing the quality of the implementation of the NMCR cycle at hospital level through a systematic methodology.9 This study aimed at evaluating the quality of the NMCR implementation in five countries of Eastern Europe and Central Asia, using WHO checklist, to identify common strengths and weaknesses among different settings.
Material and methods
Population and setting
The assessment was conducted in Armenia, Georgia, Latvia, Republic of Moldova and Uzbekistan between June 2015 and October 2016. Countries were chosen based on the following criteria: (1) activities planned by the Ministry of Health (MoH) included a quality assessment of the NMCR; (2) there was a request for technical assistance from WHO or UNFPA.
In all of the countries the NMCR approach was introduced following WHO methodology.9 The year of NMCR introduction differed among countries (table 1).
The number of facilities visited in each country depended on the total number of hospitals implementing the NMCR cycle: in Armenia, Georgia and Latvia all facilities implementing the NMCR were visited; in Moldova and Uzbekistan, where a large number of maternity units are implementing the NMCR, a sample was selected in agreement with the MoH and the national NMCR coordinator/s, following a geographical criteria (ie, so that different regions were represented) and including different types of hospitals. Overall, 23 maternity units were visited in the five selected countries (table 1).
Data collection
Each facility was visited for at least the duration of a whole day by two independent external experts with long-term experience in NMCR implementation. The international team was joined by the national assessors, experienced in NMCR implementation at local level. The team was under the leadership of one international assessor (AB), who participated in all hospital visits, with the objective of ensuring standards procedures in all assessments.
The assessment was carried out using a checklist developed by WHO to evaluate the quality of the NMCR cycle at hospital level (online supplementary table S1). The checklist was developed by WHO in 2014, field tested and optimised for use in early 2015.9 The methodology for the quality assessment is fully described in a WHO manual.9 Briefly, the checklist includes 50 items, grouped in 11 domains. The sources of information for the assessment include: direct observation and evaluation of one or more NMCR sessions; discussion with participants, coordinators and managers; documents from the NMCR sessions (templates and notes from the sessions); local documents (regional/local policies and guidance documents; protocols and standards for care; documents related to quality assurance, monitoring and supervision; reports on NMCR activities); national documents (national policies and guidance documents, guidelines and reports on NMCR implementation). According to WHO methodology, using WHO manual9 as source of standards, each of the 50 items was scored from 0 (totally inappropriate) to 3 (appropriate) (online supplementary table S1). For each of the 11 domains the arithmetic mean and 95% CIs among all the items in that domain were calculated. The median and the range between the first and third quartile (IQR range) were also calculated.
Supplementary file 1
In each facility, immediately after the assessment, feedbacks were discussed with the local staff and plans for improvement of the NMCR implementation were developed, using a simple matrix (online supplementary table S1).
After completing the visits to all maternity units in the country, a national restitution workshop was organised involving representatives from the hospitals, health authorities, professional organisations and partners. During the workshop, achievements and constraints were presented and underlying reasons were discussed. Recommendations for improvement were developed and synthesised in a standard predefined simple matrix (online supplementary table S1).
Ethical considerations
Activities of this observational study were initiated on request of the MoHs and carried out in close collaboration with the health authorities; ethical approval was not required. Information to hospital staff was provided by MoH representatives and local authorities. All people involved in the NMCR sessions were informed about the purpose of the visit and oral consent from the hospital staff and local coordinators and facilitators participating in the observed sessions was obtained. The review of near-miss cases was carried forward anonymously, that is, information that may have disclosed the identity of the patient, or providers of care, was not reported.9 This study did not aim at directly comparing countries or single facilities with different background, context and timelines of implementation, therefore, results of the assessment are reported in an anonymous way, according to WHO methodology.9 Detailed finding of the assessment together with feedback on how to improve quality of the NMCR implementation were provided to each facility and to each country individually.
Results
The assessment pointed out that quality of the implementation of the NMCR cycle was heterogeneous among different countries, as well as among different hospitals within the same country. Table 2 reports the results of the summary scores, for each of the 11 domains of WHO assessment checklist.
Overall, the first part of the audit cycle (steps 1–6 in table 2, ie, from case identification to case analysis) was on average fairly well performed in all countries (mean score 2.00, 95% CI 1.94 to 2.06), with the exception of the domain ‘inclusion of users’ views’ which was poorly implemented in most facilities (mean score 1.06, 95% CI 0.12 to 2.00). The second part of the audit cycle (steps 7–10), which involves developing appropriate recommendations, implementation of the recommendations, follow-up, documentation and dissemination of results within the facility and the country, was on average poorly performed in all countries (mean score 1.20, 95% CI 0.93 to 1.46). In particular, the domain 11 ‘ensuring quality in the NMCR cycle’, which implies a process of periodical quality assessment, development of recommendation for quality improvement and related actions, was overall substandard (mean score 0.66, 95% CI 0.05 to 1.28), with the exception of country E, where regular monitoring and supervision was carried out by a team that included national and international members.
In each country it was possible to identify at least one ‘champion’ facility, where quality of the NMCR cycle had only minor deficiencies (A-H3, B-H4, C-H1, D-H3, EH1 and H2). On the other hand, in a few facilities (A-H2; B-H1 and H3; DH6) most of the areas assessed were judged as ‘totally inappropriate’.
In some facilities examples of good practices were also observed for domains that were on average implemented on a substandard level at a country level. For example, despite inclusion of users’ views being substandard in most facilities in countries B and D (mean scores 1.11, 95% CI 0 to 2.22 and 0.61, 95% CI 0 to 1.48, respectively) single facilities reached good scores (B-H4 had a score of 3 and D-H3 had a score of 2), being able to regularly interview women and incorporating their views in the development of recommendations to improve hospital care (table 2).
On average, quality of the implementation of NMCR was on a higher level in country E, where evaluation scores pointed out that there were only few weaknesses in implementation compared with other countries (mean score 2.12, 95% CI 1.84 to 2.39).
Table 3 summarises main common strengths and weaknesses in the quality of the NMCR implementation, as divided in three categories: (1) those mostly related to technical aspects, (2) those predominantly of organisational nature and (3) those related to the attitude towards the NMCR. The main technical strength was that, beside the existence of appropriate technical skills in the methodology, most facilities developed several recommendations that were achievable, realistic, time-bound and with a potential impact on the quality of care. Although recommendations were not always well documented (thus resulting in low scores under domain 10), gaps in reporting results did not always indicated actual gaps in implementation, and in many cases several recommendations were actually implemented. This was a common observation in country B, where recommendations were poorly recorded, but several actions to improve quality of care—such as setting up emergency kits and related protocols, and introducing the Modified Obstetric Early Warning Score chart14—were actually implemented. Among strengths in organisational aspects, the most important was that NMCR was regularly held, and staffing at all levels, including midwives, participated. Main strengths in attitude included the endorsement and application of the basic principles of the NMCR (confidentiality, openness, respecting diverting opinions and avoiding blame).
Main gaps in technical aspects were: inappropriate case reconstruction, case analysis not getting to the ‘real point’ and not using a ‘why-but-why’ approach (ie, discussion of underlying causes) and recommendations not being fully SMART (specific, measurable, achievable, realistic, and time-bound).15 Main gaps of organisational nature were: lack of continuity in the role of facilitator/coordinator, lack of proper dissemination of the results (ie, circulation of information within the facility level and at national level on how many and what type of recommendations were developed) and lack of follow-up on previous recommendations. Major gaps in adopting the background philosophy and principles of the NMCR were observed in some facilities such as: lack of respect for other people’s opinion, persistence of blaming and judging others rather than using the NMCR cycle to discuss and improve ways of working and insufficient involvement of mid-level staff. Lack of inclusion of the users’ view, which was a frequent observation, was reported to be due to the lack of trained interviewers, and this was interpreted as not merely an organisational gap, but also as a problem in attitude of the health providers, that is, lack of understanding the importance of taking into account the women’s point of view. Finally, common to most facilities, there was insufficient monitoring and evaluation, and lack of a quality assurance mechanism. In most cases this was due to deficiencies in establishing and efficiently running a NMCR coordination system at national level.
Recommendations developed by local stakeholders during the national restitution workshops were setting specific. Nevertheless, there were several similarities. The most frequent/relevant recommendations developed for implementation at different levels—hospital level, national level, WHO and development partners—are reported in table 4.
Examples of the observed impact of the NMCR on quality of care at facility level are reported in online supplementary table S2. Despite progress was often poorly reported both in the hospital and in national reports, several achievements could be observed. These included improved use of national clinical guidelines, development and use of local protocols and standards of care, better availability and organisation of emergency services, improved autonomy of midwives and positive dynamics such as improved team working.
Discussion
This study aimed at evaluating the quality of the NMCR at hospital level in selected countries within WHO European Region using a standardised checklist and methodology. Overall the assessment pointed out that the practise of reviewing near-miss cases at hospital level is currently ongoing in all countries included in this study; however, both coverage and quality of the implementation of the NMCR cycle are heterogeneous. Overall, while the first part of the audit cycle (from case identification to case analysis) was fairly well performed, with the exception of the ‘inclusion of users’ views’, the second part of the audit cycle (developing recommendations, implementing them and ensuring quality) was in general poorly performed. Gaps in implementation were both of technical, organisational and attitudinal nature.
These findings are not entirely surprising. Previous, although less systematic, evaluations in the same geographical area pointed a series of challenges7 8 11 12 in effectively implementing the review of near-miss cases at facility level. Beside technical and organisational challenges, the successful implementation of clinical audits such as the NMCR often calls for a major change in staff’s attitude.7 8 11 12 In the countries assessed, especially in the ex-Soviet countries, the successful implementation of the NMCR aims at moving away from a ‘traditional’ system of carrying forward clinical audits, where blame and punishment were the routine, subjective judgement were the rule and audit involved only doctors, while midwives, other mid-level staff and service users had no voice.7 8 11 12 The ‘traditional’ audit system mainly resulted in punishing single individuals, rather than at looking to the health system failures and finding solutions at organisational level.7 8 11 12 Changing practices involved building knowledge and skills together with a drastic shift in attitude. Given these substantial constraints, the successful implementation of the NMCR at least in one country (country E) and in several champion maternity units in other countries, must be seen as a positive achievement, proving that NMCR can be successfully implemented in different settings.
This paper reports the quality of the NMCR implementation in middle-income countries (Armenia, Moldova, Uzbekistan are lower middle income countries and Georgia is an upper middle income country), where the NMCR was carried forward with relatively limited resources. Findings of this assessment cannot be generalised to other high-income countries of WHO European Region, such as UK, Norway and the Netherlands, where the practice of reviewing maternal near-miss cases has been institutionalised, with major efforts on creating coordinating mechanisms.16–18 However, it must be acknowledged that the review of near-miss cases at facility level is still not a routine practice in many European countries. We were unable to identify any study reporting on a standard-based assessment of the quality of the NMCR from any country of WHO European Region.
Interestingly, findings of this study suggest that quality of the implementation of the NMCR cycle is not strictly associated to the duration of the implementation. However, it is also true that adequate time is needed for implementation, and completing a pilot phase in a country cannot take <18–24 months from the first technical workshop. In this regard, it must be acknowledged that country B started piloting just 6 months before the quality assessment; therefore, observed results in this country can be interpreted as satisfactory given the short time frame.
The high heterogeneity in results within the same country (such as in the case of countries A, B and D) suggests that quality of the NMCR implementation depends, to a large extent, from hospital factors, including staff’s commitment, managerial support and local coordination. These results are in line with a systematic review on facilitators and barriers to effective implementation of NMCR cycle, pointing out that hospital factors (good leadership), together with a system of coordination (which often includes external support), are key enablers for effective NMCR implementation (Lazzerini, Ciuch, Covi, et al. Submitted, 2017).19
This assessment pointed out that, despite WHO recommends conducting an interview with the women/her family for each near-miss case, inclusion of women’s view was still substandard in many of the assessed facilities. However, some facilities (B-H4 and D-H3) reached good scores even when this domain was problematic at a country level (table 2). In WHO framework, ‘experience of care’ is one of the two key components of quality of maternal and newborn healthcare, along with ‘provision of care’.1 2 The views of women and their families can provide relevant information on aspects related to case management, including important details on what happened, such as organisational issues, communication issues and respectful care. In a study in Moldova, it was observed that the implementation of NMCR improved attitude towards patients,20 while in Kazakhstan it successfully improved patients’ satisfaction.21 22
This study points out that quality in the reporting on the NMCR activities was overall low. WHO manual now provides a series of templates to facilitate a uniform reporting.9 Sustained monitoring and evaluation based on appropriate reporting, as well as periodical quality assessments should be part of a strategy to achieve quality in the NMCR implementation.
This paper has the merit of reporting the actual state of implementation of NMCR in a real setting and not in a study setting (where usually a limited number of facilities is involved for a limited period of time, with dedicated human and financial resources). Another strength of the study is that the evaluation was carried out in a systematic way using a predefined standardised tool and methodology, aiming at evaluating all key aspects that contribute to overall NMCR quality (online supplementary table S1).9 To our knowledge, no other previous similar systematic evaluations have been performed.
We acknowledge that the scoring system used by the checklist may be open to some subjectivity. However, this scoring system is similar to others extensively used by WHO in the last 15 years for systematic, standard based, quality assessments and it proved to be able to capture key elements of quality of the implementation in both pragmatic and research settings.23–27 No other validated tool or scoring system exists to assess quality of the NMCR. The checklist and its score system were field tested before use, until when they were considered satisfactory covering all key aspects of quality of NMCR.9 The score is attributed by a team of experts, thus reducing subjectivity of the single individual in the evaluation.9
As a second limitation, we acknowledge that in two out of the total five countries (Moldova and Uzbekistan), the sample was selected based on MoH indications (non-probability sampling), and one cannot exclude a selection bias towards the better performing institutions. However, we emphasise that the main purpose of the assessment was to create an opportunity at national level, to discuss quality of the NMCR and to develop recommendations for improvement. Subsequent assessments could extend the evaluation to other facilities and monitor progress in specific areas.
Based on the results of this study, in the future, more efforts should be put in evaluating the quality of the implementation of NMCR on a regular basis. More implementation studies should explore interventions aiming at improving quality of the NMCR implementation in different settings.
The objective of this study was not evaluating the impact of the implementation of the NMCR, but rather the quality of the process. Nevertheless, several achievements could be observed (online supplementary table S2), despite this type of information was not consistently available. These results are in line with other studies28–41 and a systematic review reporting that NMCR is an effective strategy in improving quality of care when measured against predefined standards and it may even significantly reduce maternal mortality in high burden countries (Lazzerini, Richardson, Ciardelli, et al. Submitted, 2017).
Conclusions
Ensuring high quality in the implementation of the NMCR may be difficult in countries of Eastern Europe and Central Asia, but achievable. In the future, more efforts should be put in evaluating the quality of the implementation of NMCR on a regular basis, capitalising from these lessons and preventing and mitigating common barriers that hamper successful implementation. The availability of a new manual on how to implement and to monitor the NMCR at facility level, and of a standard methodology for assessing quality of the NMCR, as well as templates for reporting,9 may facilitate this process.
Acknowledgments
We would like to thank the local NMCR country coordinators, the hospitals staff, WHO regional and country staff and development partners who collaborated in the organisation of the assessment missions. We thank Sonia Richardson for language revision in this paper.
References
Footnotes
Contributors AB and ML conceived the study, analysed the data and wrote the first draft of the paper. AB, SH, HK, SB, SI, MJ, ID, GM and GL collected data and contributed to the final draft of the paper. GL and GM contributed by procuring funds. All authors contributed to the final version of the paper.
Funding The assessment was supported by WHO Regional Office for Europe and UNFPA Regional Office for Eastern Europe and Central Asia.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional details on the country assessments can be obtained from the first author.