Objectives To perform an external validation to assess the usefulness of the Maternal Severity Index (MSI) in predicting maternal death among women with potentially life-threatening complications during pregnancy or childbirth.
Design Prospective observational study.
Setting A tertiary referral centre in southeastern India.
Participants 1833 women with potentially life-threatening complications identified using the WHO criteria.
Predictor assessed MSI calculated based on the severity markers of the WHO criteria for maternal near-miss.
Primary outcome Maternal death.
Statistical analysis Receiver operating characteristics (ROC) curve analysis was performed to assess discriminative performance, and agreement between expected and observed deaths was plotted to determine calibration.
Results The incidence of severe maternal outcomes was 10 per 1000 live births. There were 57 (151 per 100 000 live births) maternal deaths during the study period. Maternal Severity Score was significantly higher among those who died (2.8±1.3 vs 2.0±1.2, p<0.001). The mean MSI value was 1.03% (95% CI 0.7% to 1.2%). ROC curve analysis showed good discrimination (AUC(Area Under the Curve): 0.962, 95% CI 0.952 to 0.970); however, overfitting was seen with higher probabilities. The standardised mortality ratio (SMR) was 0.02 (95% CI 0.01 to 0.02), indicating good quality of care.
Conclusions The MSI has good discriminative performance in distinguishing who succumbs to life-threatening complications, but needs recalibration to avoid overfitting. SMR of less than 0.5 indicates fewer than expected deaths, suggesting good quality of care in reducing maternal mortality in the study population.
- Maternal medicine
- PUBLIC HEALTH
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
Assessment of the discriminative performance of the score using receiver operating characteristics curve and calibration by calibration plot may be considered a strength of the study.
Prospective recruitment of cases limits recall bias.
Overfitting of the model noted in the study may be attributed to the relatively low primary event rate in the population.
Use of severity markers of the WHO criteria may limit generalisability to settings with limited laboratory and care facilities.
A life-threatening event (severe maternal outcome) during pregnancy or childbirth can result in maternal death, or a near-miss among those who survived.1 Maternal deaths have shown a declining trend globally, from 385 in 1990 to 216 in 2015 per 100 000 live births (44% reduction), even as the target of 75% reduction was missed following the various initiatives under the Millennium Developmental Goals.2 3 Maternal mortality remains one of the major health indicators under the Sustainable Developmental Goals.2 4 The WHO has published directives aiming to achieve the maternal mortality target of less than 70 per 100 000 live births by 2030 under the Ending Preventable Maternal Mortality strategies.5
With the declining rates of maternal mortality worldwide, many more survive and develop sequelae later in their lives. In 2011, the WHO recommended criteria with 25 severity indicators for identifying a life-threatening condition.6 7 Scoring systems such as Acute Physiology and Chronic Health Evaluation (APACHE), Sequential Organ Failure Assessment (SOFA), etc, have been used to evaluate the severity and outcome of critically ill patients.8 9 The Maternal Severity Index (MSI), a tool developed from a multicentre study, could predict the risk of mortality in pregnant women with life-threatening complications.6 The average of MSI can aid (1) in health impact evaluation (as part of a quality-of-care assessment) of a hospital or health facility and (2) in resource allocation by determining the level of complexity and severity of cases, and allow over-time as well as interhospital/regional comparisons.6 10 Limited studies are available in the literature validating the usefulness of the MSI, especially from low-income to middle-income countries.11 12 We aimed to perform an external validation of the MSI in predicting maternal deaths among pregnant women with potentially life-threatening complications during pregnancy or childbirth.
Study design and setting
This study was based on data collected for a primary study assessing the incidence of maternal near-miss and its impact on maternal health at 12 months among women with potentially life-threatening complications from May 2018 to April 2021. They were enrolled from the intensive care unit, high-dependency unit, eclampsia room (special care room for managing patients with pre-eclampsia/eclampsia) and labour ward of the Women and Child Hospital attached to the Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry, India. The hospital serves as a tertiary referral centre for high-risk pregnancies, mainly from the rural population of the Union Territory of Puducherry and the neighbouring districts of Tamil Nadu. The hospital manages 17 000–18 000 births a year.
Pregnant women over 18 years of age admitted with a potentially life-threatening condition, defined by the WHO criteria as a condition that can threaten a woman’s life during pregnancy and labour and after termination of pregnancy, were enrolled.1 13 These included (1) haemorrhagic disorders such as placental abruption, placenta previa, postpartum haemorrhage, ectopic pregnancy and ruptured uteri; (2) hypertensive disorders such as severe pre-eclampsia, eclampsia, hypertensive urgencies and HELLP (haemolysis, elevated liver enzymes and low platelet) syndrome; (3) other systemic disorders such as pulmonary oedema, seizures, sepsis shock, thrombocytopaenia (platelet count <100×109/L) and thyroid crisis; and (4) management indicators such as blood transfusions, central venous access, hysterectomy or surgical intervention.3 All eligible women fulfilling the study criteria admitted in the 3-year study period were enrolled in the study.
Informed consent was obtained at the time of enrolment in the primary study. The research staff noted the sociodemographic information of each woman with potentially life-threatening conditions, including age, level of education, socioeconomic class, medical and obstetric history, and reports of the laboratory investigations performed. Socioeconomic classification based on per capita income was used as follows: upper high (7533 rupees and above), high (3766–7532 rupees), upper middle (2260–3765 rupees), lower middle (1130–2259 rupees) and poor (below 1130 rupees).14 Details of care received in the hospital until discharge, details on labour and delivery, and neonatal outcomes were also collected.
Predictor variables in the MSI model
All pregnant women included in the study were assessed for presence of the 25 severity markers of the WHO near-miss criteria, representing a life-threatening event/complication. According to the WHO criteria, women who survived life-threatening complications were considered a near-miss.1 A score of 1 was given to presence of each marker, and the total score of the 25 severity markers indicates the Maternal Severity Score (MSS).6 The worst parameters in the initial 48 hours after admission to hospital/enrolment after developing potentially life-threatening complications were used to calculate individual patient severity index. The MSI calculated the probability of maternal death for each patient using the equation from the study of Souza et al6:
where x1 is the number of life-threatening complications, x2 is the life-threatening condition identified in the first 24 hours of hospital stay, x3 is severe pre-eclampsia, x4 is cancer, x5 is any marker of cardiovascular failure (pH of 7.1, use of continuous vasoactive drugs, cardiac arrest and cardiopulmonary resuscitation), x6 is any marker of respiratory failure (gasping, PaO2/FiO2 of 200 mm Hg, and intubation and ventilation not related to anaesthesia), and x7 is hysterectomy.
Standardised mortality ratio (SMR) was calculated using the Excel file provided in the study by Souza et al.6 SMR is the ratio between observed maternal mortality risk and predicted maternal mortality risk:
where the predicted death is calculated by multiplying MSI by population size.
Low SMRs suggest good performance of care (ie, observed mortality lower than expected for the level of severity), whereas high SMRs suggest poor performance of care.15
Outcome and statistical analysis
The primary outcome was the occurrence of maternal death during hospital stay. The model’s performance (ie, MSI) in predicting maternal death was evaluated. Pregnant women with details of all the predictor variables of interest were included in the analysis. Discrimination was assessed by estimating the C-statistic with its 95% CI (where a value of 1.0 represents perfect discrimination and 0.5 reflects no discriminative ability). C-statistic <0.60 indicates poor discrimination, 0.60–0.75 means moderate discrimination and >0.75 is deemed acceptable discrimination.16 The calibration plot assessed the agreement between the observed and the predicted risk of maternal death. Perfect calibration is represented as a 45° straight line with a slope of 1 and an intercept of 0.16 17 A two-sided p value <0.05 indicates statistical significance. Statistical analysis was performed using STATA V.17.0.
Patient and public involvement
There was no patient and public involvement.
Data of 1833 pregnant women with potentially life-threatening conditions among 37 590 live births during the study period, as defined by the WHO criteria, were included in the analysis. Among them, 380 (10 per 1000 live births) developed one or more life-threatening conditions (severe maternal outcome), with 57 (151 per 100 000 live births) dying from these complications. Of the women, 323 (8.6 per 1000 live births) survived the life-threatening event, forming the group with maternal near-miss. Most developed one or more severity markers and presented during the antenatal period. Caesarean deliveries were higher among those with severe maternal outcomes compared with those who did not develop severe outcomes. Table 1 shows the sociodemographic characteristics of the study population.
Among those with severe maternal outcomes, the majority had one or more haematological severity markers, with blood transfusion of 5 or more units being the most common marker. The distribution of women developing the severity markers of the WHO criteria is shown in table 2. Nearly half of the pregnancies with severe maternal outcomes had hypertensive disorders (figure 1). Sepsis and postpartum haemorrhage were the most common causes of maternal mortality (figure 2).
The average MSS was found to be 2.1 (95% CI 2.1 to 2.2) in the study and was significantly high among those who died (2.8±1.3 vs 2.0±1.2, p<0.001). The mean MSI value in the study population was 1.03% (95% CI 0.7% to 1.2%), and the SMR was observed to be 0.15 (95% CI 0.11 to 0.20). Table 3 shows the year-wise incidence of severe maternal outcome (SMO) and SMR during the study period.
As shown in figure 3, the area under the receiver operating characteristics (ROC) curve is 0.962 (95% CI 0.952 to 0.970), which indicates good discriminative performance of the MSI in distinguishing those who are at risk of maternal mortality from those who are not. A comparison of predicted probability and observed maternal mortality rates using a plot (figure 3) suggested that the predictions by the model significantly deviate from the observed rates at the highest probability, suggesting the model’s overfitting (p=0.002).
The incidence of severe maternal outcomes was 10 per 1000 live births, and the maternal mortality ratio during the study period was 151 per 100 000 live births. Majority of the women presented with severe outcomes in the antenatal period, and most had one or more markers belonging to the intervention category of the WHO criteria. ROC curve analysis of the MSI in predicting maternal death showed good discrimination (AUC (Area Under the Curve)=0.9620), but the calibration showed overfitting at higher probabilities. An SMR of less than 0.5 indicates less than expected deaths, suggesting the excellent quality of care in reducing maternal mortality. There was an increase in the SMR in the third year compared with the first 2 years.
Various clinical prediction rules, such as APACHE, SOFA, etc, are used to evaluate the severity and outcome of patients admitted to critical care units.8 9 MSS and MSI represent systems explicitly developed among pregnant women with life-threatening complications.6 These aim to assess the severity of maternal complications and predict the risk of mortality. In their multicentric study from Brazil, Souza et al,6 who developed the MSI and MSS models, reported an AUC of 0.955 and good calibration with the Hosmer-Lemeshow test. The most common severity markers (transfusion of 5 or more units of blood and acute thrombocytopaenia), the setting (referral centres in a developing country) and the discriminative performance were similar in the index study and the Brazilian derivation of the MSI study. However, the calibration plot showing the agreement between the expected and the observed was found to have overfitting in high probability ranges. This may be due to the lower number of events and the proportion of patients with higher probabilities or higher MSS in the study.
The usefulness of MSS and MSI has been studied in few reports assessing the quality of care and the performance of centres providing services to pregnant women with complications.10 11 15 18 The average MSS of 2.1 may indicate severity case mix, suggesting the human and hospital resources required for management. MSI can be used to assess the impact of a health intervention in a selected population and indicates the quality of care provided in the facility.6 As observed in figure 1, even as the number of pregnant women with hypertensive disorders was higher, most of them survived, probably due to a better quality of care. Strict monitoring, protocol-based administration of magnesium sulfate and timely delivery might have led to this low number of pregnant women succumbing to hypertensive complications. An SMR calculated from the mean MSI score of 0.15 indicates a very high performance, showing good quality of care in the facility, with more lives saved following presentation of pregnant women with life-threatening complications.6 15 The change in SMR (table 3) in the third year, during the COVID-19 pandemic, indicates the difference in the pattern of cases, as admission was restricted to those with COVID-19 during pregnancy and others with more severe complications.
This was a study conducted at a single centre, catering primarily to the rural population, to validate the use of a clinical prediction rule in the setting of a middle-income country with similar management protocols during the period, limiting selection bias, which could be a strength of the study. Unlike most of the earlier studies, the data were collected prospectively by the research team, which would have also reduced recording or recall bias. Another strength is the assessment of discriminative performance using ROC curve and the agreement of observed and expected probabilities using a calibration plot. Most earlier validation studies have only performed ROC analysis and no calibration.6 10 11 15 18 The Hosmer-Lemeshow test used in the earlier reports compared with the calibration plot used in the present study is observed to fare worse in assessing the model’s fit due to dependence on arbitrary groupings of patients.19 The main limitation was the fewer women who died following the life-threatening event, especially among those with the higher probability calculated with MSI. Even though this could be seen as due to better quality of care in the centre, it may have affected calibration assessment.
Any women who arrive late to a facility with a life-threatening event (SMO) are probably already severe and in a dying condition, and are at a higher probability of death even if they reach a tertiary facility that is fully equipped and with trained personnel.10 The SMR calculated based on the MSI may be too low for a centre with a high proportion of low-risk deliveries, which may not reflect the care in the centre.10 If the denominator is the frequency of pregnant women with potentially life-threatening conditions, it may be more likely to reflect the care given in the centre. Future studies are needed to assess the usefulness of such a change in calculation. As the current model does not take into account the intervention provided that can alter the outcome, a recalibration of the same model or the development of a new one is required, which was not attempted in the present study owing to limitations in the event rate (maternal death) and sample size.
Recent studies have highlighted the shortcomings of using the WHO near-miss tool in low-income to middle-income settings, where resources are limited. This limits the generalisability of the findings to settings where resources are limited. Recognising these limitations in a low-resource setting, where laboratory and intervention facilities are limited and with high community or home delivery rates, a global maternal near-miss criterion was proposed as an alternative.20 21
In conclusion, the MSI has good discriminative performance in distinguishing who succumbs to life-threatening complications, but with overfitting at higher probabilities. An SMR of less than 0.5 indicates less than expected deaths, suggesting the excellent quality of care in reducing maternal mortality. With the declining trend in maternal mortality ratio, a multicentric study with a larger sample size needs to assess the model and perform recalibration; incorporating the effect of life-saving interventions is required.
Data availability statement
Data are available upon reasonable request.
Patient consent for publication
This study involves human participants and was approved by the Institute Ethics Committee (Human Studies), Jawaharlal Institute of Postgraduate Medical Education & Research, Puducherry, India (approval number: JIP/ IEC/2013/3/173, dated 25 September 2013 and renewed on 25 April 2017). Participants gave informed consent to participate in the study before taking part.
Contributors AK, DMB and DKM conceived the study. All authors contributed to the design. AK, DMB, SSK and DKM carried out the data collection and guarantee data integrity. AK performed the statistical analyses. SSK and DKM reviewed the analysis. AK and DMB wrote the first draft. All authors contributed to revision and finalisation of the manuscript. AK (corresponding author and gurantor) accepts full responsibility for the work and guarantees all aspects of reliability and freedom from bias of the data presented and their discussed interpretation.
Funding The primary research project was supported by funds from the Indian Council of Medical Research, New Delhi, India (RFC no: RBMH/Adhoc/5/2017-18).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.