You are here

  • Home
  • Archive
  • Volume 13, Issue 5
  • Validating the InterVA-5 cause of death analytical tool: using mortality data from the Comprehensive Health and Epidemiological Surveillance System in Papua New Guinea

Article Text

Article menu
Global health
Original research
Validating the InterVA-5 cause of death analytical tool: using mortality data from the Comprehensive Health and Epidemiological Surveillance System in Papua New Guinea
  1. Bang Nguyen Pham1,
  2. Norah Abori1,
  3. Seri Maraga1,
  4. Ronny Jorry1,
  5. Gasowo S Jaukae1,
  6. Vinson D Silas1,
  7. Tess Aga1,
  8. Tony Okely2,
  9. William Pomat1
  1. 1Population Health and Demography, Papua New Guinea Institute of Medical Research, Goroka, Eastern Highlands, Papua New Guinea
  2. 2School of Health and Society, University of Wollongong, Wollongong, New South Wales, Australia
  1. Correspondence to Dr Bang Nguyen Pham; pnbang2001{at}yahoo.com

Abstract

Objective InterVA-5 is a new version of an analytical tool for cause of death (COD) analysis at the population level. This study validates the InterVA-5 against the medical review method, using mortality data in Papua New Guinea (PNG).

Design and setting This study used mortality data collected from January 2018 to December 2020 in eight surveillance sites of the Comprehensive Health and Epidemiological Surveillance System (CHESS), established by the PNG Institute of Medical Research in six major provinces.

Methods The CHESS demographic team conducted verbal autopsy (VA) interviews with close relatives of the deceased, who died in communities within the catchment areas of CHESS, using the WHO 2016 VA instrument. COD of the deceased was assigned by InterVA-5 tool, and independently certified by the medical team. Consistency, difference and agreement between the InterVA-5 model and medical review were assessed. Sensitivity and positive predictive value (PPV) of the InterVA-5 tool were calculated with reference to the medical review method.

Results Specific COD of 926 deceased people was included in the validation. Agreement between the InterVA-5 tool and medical review was high (kappa test: 0.72; p<0.01). Sensitivity and PPV of the InterVA-5 were 93% and 72% for cardiovascular diseases, 84% and 86% for neoplasms, 65% and 100% for other chronic non-communicable diseases (NCDs), and 78% and 64% for maternal deaths, respectively. For infectious diseases and external CODs, sensitivity and PPV of the InterVA-5 were 94% and 90%, respectively, while the sensitivity and PPV of the medical review method were both 54% for classifying neonatal CODs.

Conclusion The InterVA-5 tool works well in the PNG context to assign specific CODs of infectious diseases, cardiovascular diseases, neoplasms and injuries. Further improvements with respect to chronic NCDs, maternal deaths and neonatal deaths are needed.

  • public health
  • statistics & research methods
  • epidemiology

Data availability statement

Data are available in a public, open access repository. Data are fully accessible at https://datadryad.org/stash/dataset/doi:10.5061%2Fdryad.6wwpzgn0t. The datasets used in this study are available from the corresponding author on reasonable request. The corresponding author has full access to all the data used in this study and had final responsibility for the decision to submit the study for publication.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2022-066560

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • This validation study used data from deaths that occurred in the communities, collected by community-based surveillance teams via verbal autopsy (VA) interviews with close relatives of the deceased, using the WHO 2016 VA interview instrument.

    • The InterVA-5 tool assigned specific causes of death (CODs) for more than 900 deceased people, as per the International Classification of Diseases 10th Revision.

    • CODs were independently reviewed, verified and certified by the medical team, using the medical review method as the ‘gold standard’ in mortality audit and COD certification.

    • Mortality data were recorded from eight surveillance sites, not representative of all deaths in Papua New Guinea or of all deaths that occurred in the catchment areas of the Comprehensive Health and Epidemiological Surveillance System; as such, caution should be exercised in interpreting our findings, especially for those CODs with small numbers of observations.

    • Large volumes of migrants moved in and out of the surveillance sites prior to the COVID-19 pandemic, and lockdowns and isolation occurred during the pandemic, which affected the data collection and data quality.

    Introduction

    Nearly half of all countries fail to meet the United Nations’ standards for death registration (90% coverage), and 65% of the world’s population are lacking high-quality mortality data that can be used for cause of death (COD) analysis, hindering the effective development of health policy, social planning, and monitoring and evaluation.1

    Papua New Guinea (PNG) is located just south of the equator, 160 km north of mainland Australia in the South Pacific region, and consists of 22 provinces with a population of approximately 8.7 million in 2020.2 PNG has undergone an epidemiological transition as a result of the recent socioeconomic development in the country.3 The Civil Registration and Vital Statistics system is considered ‘gold standard’ for monitoring and reporting COD in the population.4 However, this system is underdeveloped in PNG, particularly in rural areas.5 The National Health Information System (NHIS) collects mortality data with a focus on deaths that occurred in health facilities, mostly at tertiary hospitals. This results in biased and incomplete mortality data for analysing CODs in the population.6

    The Comprehensive Health and Epidemiological Surveillance System (CHESS) is a new generation population-based surveillance system with an electronic population database, developed by the PNG Institute of Medical Research (PNGIMR). CHESS was based on the integrated Health and Demographic Surveillance System, which started in 2011, and replaced by CHESS in 2018.7 CHESS was financially supported by the PNG government and approved as an integral part of the PNG National Medium Term Development Plan 2018–2022.6 PNGIMR’s CHESS is 1 of 49 surveillance centres in the INDEPTH Global Surveillance Network, which collects high-quality mortality and morbidity surveillance data in diverse settings in low/middle-income countries.1 Mortality data collection has been integrated as part of the ongoing CHESS activities of PNGIMR.

    Verbal autopsy (VA) interview and automated COD analysis methods such as SmartVA have been used in PNG and Solomon since 2010s,3 8 based on a reasonably representative sample of the population living in the surveillance sites. Deaths that occurred at secondary and tertiary health facilities were medically certified with COD diagnosed by medical officers. However, death certification services are often unavailable in rural areas, where an estimated 85% of deaths occur.3 Since 2005, a standardised approach to VA interviews has been developed and applied to studies to collect mortality data in several countries, including PNG.9–12 Collecting mortality surveillance data from the communities is particularly challenging due to the significant geographical, cultural and political diversity of PNG,13 where sorcery is common and often referred to as a COD whenever sudden death has occurred. ‘Old age’ is also commonly referred to as a COD in communities. Lessons learnt with implementing mortality surveillance activities in PNG suggested that logistical arrangements and incentives should be considered as strategies to maximise completeness of mortality data, and health workers have great potential to improve the completeness of death registration in PNG.14

    Lack of quality mortality data and inadequate analytical methods are the main reasons for poor data on CODs among the population in PNG.15 VA by a medical review is necessary, but it is almost impossible to conduct for all deaths due to inadequate financial and human resources. Automated COD analytical tools should be considered. There are currently three families of automated VA models of relevance to the WHO 2016 standard, including InterVA, InSilico VA and Tariff.

    The InterVA-5 is a new computer program analytical tool that was built substantially on the basis of InterVA-4 model. The InterVA-5 was published in 2019. The tool has been tested in different settings. The method offers potential benefits over previous methods. The InterVA-5 includes important new concepts and updates. This tool aligns with the WHO VA instrument1 and other existing VA data collection tools. The design of InterVA-5 directly corresponds to WHO 2016 and also incorporates compatibility with WHO 2012 and InterVA-4, as well as coherence with Tariff-2 and Smart VA model. The InterVA-5 was designed to process VA data, using variables to generate standardised CODs. The tool can be used as an independent or complimentary tool for COD analysis at the population level. CODs assigned by the InterVA tool can be triangulated with COD data from a medical review. InterVA-5 empowered community health workers in identifying COD during the COVID-19 pandemic in Indonesia.16 As more new primary mortality data collected under the WHO 2016 VA instrument became available, there are opportunities to further assess the validity of this tool.

    Smart VA was used in the previous study on CODs conducted in PNG in the period 2011–20153. InterVA-5 was selected for this validity study as part of the PNGIMR’s commitment to improving this tool and continuing the INDEPTH Network activities in PNG.17 The InterVA-5 method was applied for the first time in PNG in 2020. Other VA analytical tools such as InSilico VA and Tariff were not included in this validity study.

    This study is the first to validate the CODs ascribed by the InterVA-5 model against the medical review, using the new surveillance mortality data from CHESS in PNG. A lack of quality mortality data and inadequate analytical methods are the main reasons for poor data on CODs among the population in PNG.15 VA by medical review is necessary, but it is almost impossible to conduct for all deaths due to inadequate financial and human resources. Automated COD analytical tools such as InterVA-5 tool should be considered, but the validity of this tool has not been evaluated in PNG yet.

    This study aimed to determine the validity of the InterVA-5 tool in the PNG context. In this study, CODs assigned by the InterVA tool were validated against the medical review method. The validation will provide insight into the use of the method, for improvement in assessing COD in PNG, and evidence for scaling up of the use of the tool in similar settings. The study addressed the following research questions:

    1. What are the consistency, difference and agreement between the InterVA-5 tool and medical review in ascribing CODs?

    2. What are the sensitivity and positive predictive value (PPV) of the InterVA-5 tool in assigning specific CODs compared with the medical review?

    Methods

    Data source

    This study used mortality data from PNGIMR’s CHESS, which was designed as a longitudinal follow-up cohort study to provide up-to-date data series on a range of key public health indicators of the populations living within the surveillance sites and the socioeconomic changes over the years. The system is a long-term effort of the PNGIMR with financial support from the PNG government to meet the country’s need for high-quality data for health and development. CHESS includes eight sentinel surveillance sites located in urban and rural areas of six major provinces: Central, Madang, Eastern Highlands Province, East New Britain, East Sepik Province and Port Moresby (POM)—the National Capital District. The CHESS methodology has been previously published.6 18 CHESS database consists of five data components: (1) household socioeconomic status and demographic data; (2) children under 5 years of age; (3) women of reproductive age 15–49 years; (4) men of working age 15–64 years; (5) morbidity data of patients seeking healthcare services at primary health facilities and mortality data of the deceased who died in the communities within the surveillance sites. These data components are interlinked by unique household and individual ID codes, which are constructed and aligned with the national coding system issued by the National Statistics Office in 2010.19

    Mortality data used in this study were extracted from the mortality data component, which were collected from the communities by the CHESS demographic team from January 2018 to December 2020 through VA interviews with close relatives of the deceased, who live in the surveillance sites, using the WHO 2016 VA instrument.20

    Deaths in the communities were identified by village-based data reporters. With access to social networks, data reporters were aware of deaths that occurred in their villages. Data reporters were advised to identify all deaths occurring in their villages, but recommended only including deaths that occurred within 2 years prior to the commencement of the study and during the data collection period for VA interviews.21 The defined range of eligible dates of death minimised recall biases and ensured no overlap between mortality data used in this study and data used in the previous study in the period 2011–2015. They had ready access to the households through regular visits and made arrangements for VA interviews at a convenient time for both the interviewer and interviewee.

    WHO 2016 VA data collection tool

    The WHO VA data collection instrument was developed, based on the consolidation and integration of various existing VA tools.22 This tool was designed to facilitate interviews with relatives of the deceased using portable electronic devices such as a tablet and smartphone. Different sets of questions are asked about clinical signs, medical records, personal and family history of the deceased in an interactive manner with skip questions that offer considerable improvement in efficiency and effectiveness.23

    The WHO 2016 VA instrument has been adapted for optimal use in the PNG context.24 The tool had a section on identification information, including the household and individual identification codes and global positioning system information, aligned with other data collection tools such as the household socioeconomic questionnaire and the morbidity questionnaire currently in use in CHESS. This allows linkage between these data components as well as other potential national data sources.7

    VA interviews were usually scheduled in the 2 weeks after the mourning period, but the organisation of VA interviews might take up to several weeks due to logistical arrangements, the availability of interviewees and transportation means. Some VA interviews required more than one visit to complete. VA interviews were conducted by the CHESS demographic team, including six national scientific officers. Most VA interviews were conducted in Tok-Pisin, the most common local language in PNG and Motu language, which was used in Central Province.

    The principal investigator of CHESS was trained on VA and COD methods in the previous COD study from 2013 to 2015. The first training on the WHO 2016 VA instrument was held by the principal investigator for CHESS staff in April 2017, followed by a refresher training session in March 2018 before the mortality data collection was scaled up across the surveillance sites. The refresher training focused on the design of the WHO 2016 VA tool, including the questionnaires, technical terminologies and definitions. The training included a practical section on recording the VA interview information in the tablets and uploading the VA data from the tablets into the CHESS database. The training also provided the data collectors with techniques for use and maintenance of the tablet, as well as in communication skills, necessary for a successful VA interview in the PNG local context.7

    InterVA-5 COD analytical method

    The InterVA-5 COD analytical method was used to identify specific CODs of each of the deceased. This computer software program can assign 64 CODs and categories, aligned with the International Classification of Diseases 10th Revision (ICD-10).25 Like the previous versions of InterVA family, the InterVA-5 tool uses a simple input format of binary questions. InterVA-5 uses a data-driven concept of a substantive response for each item, which may be ‘yes’ or ‘no’. The probabilistic modelling updates the likelihood for each COD category on the basis of the substantive responses recorded in the VA dataset. The InterVA-5 tool can assign more than one specific COD for a death with a respective likelihood (%), which adds up to 100%. In this study, only the first ascribed specific COD with the highest likelihood for each death was presented. The term ‘undetermined’ was used if no COD was programmatically identified.1 The mortality dataset used in this study was extracted from the CHESS database in an Excel spreadsheet format, using SQL/process maker, and converted into InterVA-5 input format (cvs.file) that included 353 variables and identifiers.

    Medical review to assess InterVA-5 validity

    Medical review is considered a ‘gold-standard’ method for COD analysis, using mortality data from VA interviews.26 The medical review method is reliable and verifiable for ascertaining a specific COD from VA data. In this approach, medical officers are trained in COD definition and certification. They review VA data, collate information from various sources and assign probable disease causes, which are directly leading to death. International standards and practices for COD certification such as the ICD-10 are used to code underlying CODs.27 The medical review remains the core element for quality control and validation of CODs.28

    The medical review was conducted by the CHESS clinical team, including an international medical officer (CHESS principal investigator), a national health extension officer and a national nursing officer, following the standard procedure for COD verification and certification, which have been defined in the previous study on COD conducted in PNG in the period 2011–2015.3 The medical review team members reviewed the VA interview data of the deceased independently. Since this study focused on deaths that occurred in the communities only, medical records such as hospital death certificates were not included in the review. COD diagnosis was given to each deceased person according to the gold-standard definitions.26 Medically reviewed CODs were cross-checked among the reviewers. Differentials in the COD diagnoses were discussed among the team to reach an agreement. The medical review process was conducted separately from the InterVA-5 analysis. The medical review team was blinded of the result of the InterVA-5 analysis.

    Validity of the InterVA-5 tool was assessed in terms of accuracy of the ascribed CODs. The accuracy of InterVA-5-assigned CODs was assessed in comparison with the medical review. Cross-tabulation analyses were conducted to calculate the consistency and difference between CODs assigned by the InterVA-5 tool and those diagnosed by medical review. Kappa statistics, SE and Χ2 tests were used to estimate the levels of agreement, SE and significance between the two methods. Kappa scores were categorised into three levels of accuracy: 0–0.39 (low), 0.4–0.69 (moderate) and >0.7 (high). A higher kappa score reflected a more accurate diagnosis of a COD.29

    The validity of InterVA-5-assigned CODs was evaluated by measuring the sensitivity and specificity, PPV and negative predictive value (NPV),30 using the following formulas:

    View this table:

    The sensitivity and PPV of the InterVA-5 model were calculated for every InterVA-5-ascribed COD with reference to the medically reviewed COD.

    • Sensitivity of InterVA-5 for a specific COD is the ‘true positive rate’, measured as the proportion of deaths ascribed by the InterVA-5 among the total deaths from that COD certified by the medical review method, equivalent to A/(A+C)×100.

    • PPV of InterVA-5 for a specific COD is the proportion of deaths, certified by the medical review method among the total deaths from that COD ascribed by the InterVA-5, equivalent to A/(A+B)×100.

    The higher the scores of sensitivity and PPV (in percentage), the greater the accuracy of COD assigned by the InterVA model. This validation method has been previously published. For the InterVA-5-ascribed CODs with low and moderate sensitivity and PPVs such as maternal deaths and neonatal deaths, the mortality data were further verified and the ascribed CODs were further investigated and certified by the clinical team. Due to the nature of morality data used in the study, the specificity and NPV of InterVA-5 were not calculated in this study.

    Certification of maternal causes of death

    Maternal mortality is an important public health indicator of PNG. A further medical audit was conducted by the clinical team in an attempt to confirm these maternal deaths and to certify the specific causes of maternal deaths. Maternal mortality is among the most important population health indicators as it shows the effectiveness and efficacy of the health system. Maternal mortality is recommended for monitoring and reporting in the Sustainable Development Goal 3,31 32 as well as in the PNG National Health Plan 2016–2020.33 High maternal mortality ratio (MMR) of PNG has been previously reported in PNG, ranging from 98 to 733 per 100 000 live births.34 35 The current national estimate was 171 (95% CI: 95 to 247),2 but the NHIS showed 500 per 100 000 live births.34 Understanding of health systematic and local contextual risk factors of maternal deaths would be helpful for policymakers to determine possible solutions to prevent maternal deaths, which will contribute to improvement in maternal health in PNG.

    Identification of COVID-19-suspected deaths in the community

    The first COVID-19 case was officially reported in PNG on 20 March 2020. As of the completion of the data collection of this study in December 2020, only nine COVID-19 deaths had been reported from 914 people who tested positive from PCR tests. All these COVID-19 deaths were from the General Hospital in POM.36 Little is known about COVID-19 deaths in communities in PNG. Because mortality data were collected during the COVID-19 outbreaks and the COVID-19 testing service was not available in PNG at the time of study, the study attempted to examine if any COVID-19-suspected deaths occurred in the communities during the data collection period and were captured in the mortality surveillance data.

    COVID-19-suspected deaths in the community were identified by the medical review using a defined set of diagnostic criteria, developed by the CHESS surveillance team based on the typical clinical signs and symptoms of COVID-19,37 and the individual and family epidemiological histories of COVID-19 infections.36 To facilitate and standardise the procedure of identification of COVID-19-suspected deaths in the communities, a set of six diagnostic criteria was developed by the study team and shown in online supplemental box 1.

    Patient and public involvement

    None.

    Results

    Overall distribution of mortality

    Among the 1021 deaths identified in the communities within the surveillance sites over the data collection period, 1003 VA interviews were conducted with consent, resulting in a participation rate of 98%. All VA interview data were included in the COD analysis. The InterVA-5 tool successfully assigned specific COD for 926 deaths (92%), and 77 deaths (8%) were excluded from the InterVA-5 model because of poor data quality. No specific COD was assigned to these excluded VA data. There were 31 deaths assigned as ‘undetermined CODs’ by both InterVA-5 and medical review method, accounting for 3.3% of the total deaths.

    Table 1 shows the distribution of CODs by sociodemographic characteristics of the deceased. Among 926 deceased people with assigned specific CODs, there were 93 child deaths (age at death 0–14 years), 574 adult deaths (age at death 15–64 years) and 257 elderly deaths (age at death 65+ years) (2 were missing information on age). Fewer male deaths were reported than female deaths in this mortality data (44.5% and 55.5%, respectively) across the sites, except for Madang, where female deaths were two times higher than male deaths (67% and 33%, respectively). Sex distribution of the surveillance population was 51.9% males and 48.1% females, with the male-to-female ratio of 107.9 for all the sites.38

    View this table:
    Table 1

    Distribution of deaths by age at death (in years), sex of the deceased and province, PNGIMR’s CHESS, 2020

    Consistency of InterVA-5-assigned CODs with medical review

    Online supplemental table 1 shows the level of agreement, consistency and difference between the InterVA-5 tool and the medical review for leading CODs. The overall agreement between the two methods was high with a kappa score of 0.728 (SE ±0.015; p<0.01), suggesting the high consistency between the specific CODs ascribed by the InterVA-5 tool and those diagnosed by the medical review team. However, there were some inconsistencies between the two methods. For example, among 15 CODs which were identified by the InterVA-5 model as the leading CODs among the population, only 11 CODs were confirmed by the medical review team, including pulmonary tuberculosis (TB), acute respiratory tract infections (ARTIs), HIV/AIDS, diarrhoeal diseases, acute cardiac diseases, stroke, digestive neoplasms, diabetes mellitus, respiratory neoplasms, reproductive neoplasms and road traffic accidents. In contrast, chronic obstructive pulmonary disease (COPD) was ranked as the 10th leading COD by the medical review, but not among the 15 leading CODs assigned by the InterVA-5 tool.

    The InterVA-5 method assigned 84 deaths from pulmonary TB, while the medical review identified 98 deaths from this disease. The InterVA-5 method miscategorised 14 medical review-diagnosed pulmonary TB deaths to other COD categories, including ‘other cardiac diseases’ (7 deaths), ‘other infectious diseases’ (3 deaths) and ‘other neoplasms’ (4 deaths). Similarly, the InterVA-5 correctly assigned 81 deaths from ARTIs, but misclassified 15 ARTI deaths to other diseases, that is, ‘other cardiac diseases’ (8 deaths), ‘other infectious diseases’ (5 deaths) and ‘other neoplasms’ (2 deaths).

    The InterVA-5 misclassified large numbers of deaths to three categories: ‘other and unspecified infectious diseases’, ‘other and unspecified neoplasms’ and ‘other and unspecified cardiac diseases’ against the medical review. Among 30 deaths assigned by the InterVA-5 tool as ‘other and unspecified infections’, only four deaths from this COD were matched with the medical review diagnosis, resulting in the PPV of 13.3%. According to the medical review, 10 out of these 30 deaths were related to infectious diseases (1 sepsis, 5 acute respiratory infections and pneumonia, 1 diarrhoeal disease and 3 pulmonary TB diseases), and 16 deaths were attributed to non-communicable diseases (NCDs) (including 5 neoplasms, 7 COPDs, 1 asthma, 1 acute abdomen and 2 unspecified maternal CODs). In addition, only 2 deaths from the ‘other and unspecified neoplasms’ were confirmed by the medical review out of 24 deaths classified by the InterVA-5 model under this category. Hence, the PPV of InterVa-5 for this COD was only 8.3%. Similarly, under the category of ‘other and unspecified cardiac diseases’, only 10 out of 62 deaths were certified by the medical review, resulting in the PPV of 16.1%.

    Sensitivity and PPV of InterVA-5-assigned CODs

    Table 2 shows the sensitivity and PPV of specific CODs assigned by the InterVA-5 model with reference to the medical review. For infectious diseases, three leading CODs were identified by InterVA-5 including HIV/AIDS, pulmonary TB and pneumonia. For these CODs, the sensitivity of InterVA-5 model was high, above 80%. The PPV of InterVA-5 was 100% across infectious diseases, except for the category of ‘other and unspecified infections’, where the PPV of this method was 13%.

    View this table:
    Table 2

    Sensitivity and specificity of InterVA-5 method with reference to medical review, PNGIMR’s CHESS, 2020

    For neoplasm-attributable deaths, the sensitivity of InterVA-5 was 84% for all CODs, but 58% for oral cancers and 60% for breast cancer. Although the PPV of this method was 85% for this group of CODs, it was as low as 8% for the category of ‘other and unspecified neoplasms’. Regarding emerging NCDs such as acute cardiac diseases, stroke and diabetes mellitus, the sensitivities of the InterVA-5 model ranged from 88% to 96%, with specificities of 100%.

    By contrast, the sensitivity of InterVA-5 for COPD and asthma was 35% and 14%, respectively, despite the PPV of 100%. In other words, the InterVA-5 model identified only 10 out of 28 deaths from COPD and only 1 out of 7 deaths from asthma. These InterVA-5-ascribed CODs were confirmed by the medical review. Notably, for maternal and neonatal CODs, the PPV of InterVA-5 was low (64% and 54%, respectively). For abortion and congenital malformation deaths, the PPV of InterVA-5 was even lower (50% and 15%, respectively).

    Accuracy of InterVA-5-assigned maternal and neonatal CODs

    Online supplemental table 2 shows the results of medical confirmation of maternal deaths and medical certification of specific maternal CODs. There were 11 maternal deaths identified by InterVA-5 in the previous analysis, including obstetric haemorrhage (6), abortion-related causes (2), pregnancy-induced hypertension (1), pregnancy-related sepsis (1) and other unspecified maternal causes (1), with the likelihood of above 80%, except for the category of ‘other unspecified maternal cause’ (58%). The medical review confirmed 9 maternal deaths out of 11 InterVA-5-assigned maternal deaths (81.8%), and the diagnoses of specific CODs of these maternal deaths were obstetric haemorrhage (5), abortion-related deaths (2), breast neoplasms (2), pregnancy-related sepsis (1) and other unspecified maternal cause (1). The medical review did not certify two maternal deaths assigned by the InterVA-5 method, and diagnosed breast neoplasms as the cause of these two female deaths. No other maternal death was identified by the medical review from the female adult deaths in this study. Notably, most of maternal deaths were from rural areas and in the young age group, 15–34 years, raising a public health concern over the universal access to prenatal and postnatal care services in PNG. Although the number of maternal deaths identified in this study is small and cannot be used for estimation of MMR, the figures were scientifically identified by InterVA-5 and validated by the medical review. More studies on maternal deaths using similar research methods would provide further insights into the circumstances where maternal deaths are more likely to occur.

    Among 19 neonatal deaths, InterVA-5 assigned 13 deaths due to congenital malformation, of which, 12 were from Hiri site in Central Province. As shown in online supplemental table 3, these 13 neonatal deaths were confirmed by the medical review, but this method assigned different specific CODs, including prematurity (4), fresh stillbirth (3), birth asphyxia (2), stillbirth (1) and undetermined (1). Only two neonatal deaths were diagnosed as congenital malformation by the medical review, the same as the InterVA-5 ascription.

    Identification of COVID-19-suspected deaths in communities

    Among 926 deceased people, ARTIs and pneumonia were assigned as the CODs of 81 deaths by the InterVA-5 model and 96 deaths by the medical review (see table 2). These ARTIs and pneumonia-related deaths were further investigated by the study team using the defined COVID-19-suspected death diagnostic criteria. Online supplemental box 2 shows the summary of two COVID-19-suspected deaths in the communities identified by the medical review of the mortality data collected by CHESS as of the end of December 2020. These deceased people were both above 50 years of age, had demonstrated typical clinical symptoms and signs of pneumonia such as high fever, cough and difficulty breathing prior to death, and had background diseases such as cancer and diabetes.

    Discussion

    In this study, the WHO 2016 VA interview instrument was used to collect mortality surveillance data among the population living within the CHESS surveillance sites across six provinces of PNG. The InterVA-5 COD analytical tool was used to assign CODs, which were then validated by the medical review method to confirm the assigned CODs.

    Validity of InterVA-5 COD analytical tool

    Among 15 leading CODs among the population identified by the InterVA-5 method, 12 CODs were matched with the medical review, suggesting a high level of agreement between the two methods in assigning these CODs (see online supplemental table 1). The InterVA-5 model showed high sensitivity and specificity in ascribing CODs from infectious diseases (above 80%) (see table 2). From our field observations, relatives of the deceased who died from infectious diseases often had medical records, where the information related to the deaths was written down and reflected on during the VA interviews. By contrast, the InterVA-5 had low sensitivity and specificity in assigning CODs from NCDs. The medical review noted that the deceased who were reported as having ‘chest pains’ tended to be assigned to ‘other cardiac diseases’ as the primary COD by the InterVA-5 method. Important clinical signs of cardiac diseases such as a cough, difficulty breathing and weight loss were seemingly omitted in the InterVA-5 software.

    The sensitivity and specificity of the InterVA-5 method in assigning maternal CODs were 77.7% and 64%, respectively. However, the InterVA-5 method showed relatively low specificity for abortion-related deaths (50%), suggesting further improvement in analysing this COD. Retrospective data on abortion are limited in PNG as it is often considered a highly sensitive issue. The information on maternal deaths was even more likely to be biased during the VA interviews. Our fieldwork observations suggest that respondents hesitated to discuss this COD and they tended to provide self-inference information about maternal deaths. This could be the reason explaining the low sensitivity and specificity of InterVA-5 regarding this particular COD. The confirmed maternal deaths in this study accounted for 1% of the total adult deaths, but 3% of the total female adult deaths.

    Notable is the low specificity of InterVA-5 in ascribing congenital malformation among neonatal deaths (15%). This figure is more likely due to the poor quality of VA data rather than the InterVA-5 program itself. The medical review revealed important medical information on the neonatal deaths that had been missing or not recorded properly during the VA interviews. This finding highlights the importance of appropriate training on the WHO 2016 VA instrument and the quality of VA data for COD analysis, particularly for neonatal CODs.

    The InterVA-5 showed low specificities for the categories of other and unspecified causes. For example, the InterVA-5 specificity was only 13% for the category of ‘other and unspecified infectious diseases’, 8% for ‘other and unspecified neoplasms’, and 16% for the ‘other and unspecified cardiac diseases’, suggesting an inaccuracy of the InterVA-5 model in these COD categories. The likelihood of the specific CODs provided by the InterVA-5 model was also inaccurate when many CODs were assigned a likelihood of 100%, but they turned out to be different CODs under the medical audit. The categories appeared overestimated in programming the InterVA-5. The algorithms underlying these categories could have a programmatic problem that requires further investigations to improve the performance of the InterVA-5 tool.

    Identification of COVID-19-suspected deaths in the communities

    It is noted that questions for collecting information that can be used to identify COVID-19 deaths in the communities were not included in the WHO 2016 VA instrument, and COVID-19 was not included in the InterVA-5 tool. COVID-19 testing services, including rapid antigen diagnostic and PCR tests, had limited availability at the time of the study, particularly at the community level and in the rural areas of PNG. Hence, the medical review appears the only method available to identify COVID-19-suspected deaths that occurred in the communities.

    Using these defined COVID-19 diagnostic criteria, the study team reviewed all the deaths. If a deceased person had all six criteria, a COVID-19-suspected death was reasonably sound. Two COVID-19-suspected deaths were identified from the mortality data. This finding is likely consistent with the trend of COVID-19 mortality reported in PNG at that time.36 Given no other reliable method was available to detect COVID-19 deaths in the communities, the VA methods used in this study could be used in PNG and could potentially be replicated in similar settings. Our observations also suggest that the WHO 2016 VA instrument should be modified to include individual and family epidemiological history such as travel to COVID-19 outbreak areas, contact with COVID-19 positive cases in the past, and typical clinical signs of COVID-19 such as recent loss of taste and smell, and COVID-19 testing results prior to death. COVID-19 COD should be included in the revision of the InterVA-5 COD tool. Further research into this topic appears to be a pressing need in the new normalcy brought about by COVID-19. The new version of the WHO 2022 VA instrument was recently released and included COVID-19-related questions.39

    Limitations

    CHESS is designed to provide surveillance data from sentinel sites. Hence, the provided mortality data used in this study were not representative of the entire population of the study provinces, and the CODs may not represent the mortality patterns of the entire PNG population. Mortality data were collected over 3 years and were relatively small, particularly data from POM and Madang, limiting in-depth analysis of mortality trend. Accurate diagnosis of CODs is largely dependent on the quality of mortality data. However, large volumes of migrants moved in and out of the surveillance sites during the data collection period, making it impossible to assess the completeness of collected mortality data. Refresher training on the updates and new versions of the WHO 2016 VA instrument and the InterVA-5 diagnostic tool is needed, particularly for staff with non-medical background, to further improve the completeness of mortality data and the accuracy of COD diagnosis.

    The greatest challenge to the application of the WHO 2016 VA tool in the PNG context is to ensure the users have the skills that are required to properly maintain the tablets and successfully conduct interviews in the local settings. Properly recording the death information onto the device, transferring the recorded data to the main office for processing, successfully extracting the data and linking different data tables are key to having a quality dataset for COD analysis.7 In this regard, stable connection and access to the internet with high speed are crucial to optimise the application of this tool.

    The fieldwork and data collection interrupted by lockdowns and isolation measures during the COVID-19 outbreaks in PNG had prolonged the time from the date of death to the date of completion of VA interview, potentially raising recall biases. The uncertainty surrounding the reported number of COVID-19-suspected deaths in this study must be noted as it may change considerably over time as the outbreak spread. Hence, the findings of our study should be interpreted with caution.

    Conclusion

    The InterVA-5 tool is a useful public health research method for diagnosis of COD in the population. This tool has been validated by a medical review, using mortality surveillance data from CHESS in PNG. Findings from this validity study suggest that InterVA-5 is a possible replacement for the medical review as it works well in the PNG context, and could potentially be scaled in similar settings. There is room for improvement in the PPV of the tool under the categories of ‘other and unspecified infections’, ‘other and unspecified neoplasms’, and ‘other and unspecified cardiac diseases’. Further validation of the tool is needed regarding its sensitivity in identifying deaths from COPD and asthma. The PNG government should consider using this tool for monitoring and reporting causes of death in the mortality transition in PNG. CHESS has provided a valuable data source for monitoring and reporting causes of deaths in the mortality transition in PNG.

    Data availability statement

    Data are available in a public, open access repository. Data are fully accessible at https://datadryad.org/stash/dataset/doi:10.5061%2Fdryad.6wwpzgn0t. The datasets used in this study are available from the corresponding author on reasonable request. The corresponding author has full access to all the data used in this study and had final responsibility for the decision to submit the study for publication.

    Ethics statements

    Patient consent for publication

    Ethics approval

    CHESS was granted ethics approvals from the Institutional Review Board (IRB) of Papua New Guinea Institute of Medical Research (IRB approval no. 18.05) and the Medical Research Advisory Committee (MRAC) of Papua New Guinea (MRAC approval no. 18.06). These approvals covered all the data components under CHESS, including the mortality data, which were used in this manuscript. Informed consent was sought from self-identified close relatives of the deceased. They were informed about their right to withdraw from the study at any stage.

    Acknowledgments

    We acknowledge the following individuals and organisations: community leaders; councillors and religious leaders; community members in the surveillance sites; the provincial health authorities in Central, Eastern Highlands, East New Britain, East Sepik, Madang Province and Port Moresby; and local partners of the CHESS programme: Salvation Army, Evangelical Church PNG, Lutheran Health Services, Evangelical Brotherhood Church Health Services and Catholic Health Services.

    References

    1. ↵
      1. Byass P,
      2. Hussain-Alkhateeb L,
      3. D’Ambruoso L, et al
      . An integrated approach to processing WHO-2016 verbal autopsy data: The Interva-5 model. BMC Med 2019;17:102. doi:10.1186/s12916-019-1333-6
    2. ↵
      1. National Statistics Office
      . Papua new guinea demographic and health survey 2016-18: key indicators report. National Statistics Office, 2019.
    3. ↵
      1. Gouda HN,
      2. Hazard RH,
      3. Maraga S, et al
      . The Epidemiological transition in Papua New Guinea: New evidence from verbal autopsy studies. Int J Epidemiol 2019;48:966–77. doi:10.1093/ije/dyz018
      OpenUrlPubMed
    4. ↵
      1. Lopez AD,
      2. Setel PW
      . Better health intelligence: a new era for civil registration and vital statistics. BMC Med 2015;13:73. doi:10.1186/s12916-015-0333-4
    5. ↵
      1. Adair T,
      2. Lopez AD,
      3. Queiroz B
      . Estimating the completeness of death registration: an empirical method. PLOS ONE 2018;13:e0197047.doi:10.1371/journal.pone.0197047
    6. ↵
      1. Pham B,
      2. Whittaker M,
      3. Pomat W
      . CHESS: A new generation of population health surveillance for sustainable development of Papua New Guinea. PNG Med J 2017;60:154–72.
      OpenUrl
    7. ↵
      1. Pham BN,
      2. Jorry R,
      3. Abori N
      . Comprehensive health and epidemiological surveillance system technical report: mortality surveilance in communities. Goroka: PNG Institute of Medical Research, 2020.
    8. ↵
      1. Reeve M,
      2. Chowdhury H,
      3. Mahesh PKB, et al
      . Generating cause of death information to inform health policy: implementation of an automated verbal autopsy system in the Solomon Islands. BMC Public Health 2021;21:2080.doi:10.1186/s12889-021-12180-y
    9. ↵
      1. Moir JS,
      2. Garner PA,
      3. Heywood PF, et al
      . Mortality in a rural area of Madang Province, Papua New Guinea. Ann Trop Med Parasitol 1989;83:305–19.doi:10.1080/00034983.1989.11812349
      OpenUrlPubMedWeb of Science
    10. ↵
      1. Genton B,
      2. al-Yaman F,
      3. Beck HP, et al
      . The epidemiology of malaria in the Wosera area, East Sepik province, Papua New Guinea, in preparation for vaccine trials. II. mortality and morbidity. Ann Trop Med Parasitol 1995;89:377–90. doi:10.1080/00034983.1995.11812966
      OpenUrlPubMedWeb of Science
    11. ↵
      1. Coakley K,
      2. Lehmann D,
      3. Smith D
      . The Asaro Valley surveillance unit of the Papua New Guinea Institue of medical research: methodology, demograohy and mortality. Goroka: Papua New Guinea Institute of Medical Research, 1993.
    12. ↵
      1. Lehmann D,
      2. Vail J,
      3. Vail P
      . Demographic surveillance in Tari, southern highlands Province, Papua New Guinea: methodology and trends in fertility and mortality between 1979 and 1993. Goroka: Papua New Guinea Institute of Medical Research, 1997.
    13. ↵
      1. Abdad MY,
      2. Soli KW,
      3. Pham B, et al
      . Diarrhoeal disease surveillance in Papua New Guinea: Findings and challenges. Western Pac Surveill Response J 2020;11:7–12. doi:10.5365/wpsar.2018.9.2.006
      OpenUrlPubMed
    14. ↵
      1. Hart JD,
      2. Kwa V,
      3. Dakulala P, et al
      . Mortality surveillance and verbal autopsy strategies: experiences, challenges and lessons learnt in Papua New Guinea. BMJ Glob Health 2020;5:e003747.doi:10.1136/bmjgh-2020-003747
    15. ↵
      1. Pham BN,
      2. Silas V,
      3. Jorry R
      . Marijuana-use related Homicide: A case study in Papua New Guinea. Ann Case Report 2020;14:312.
    16. ↵
      1. Siregar KN,
      2. Kurniawan R,
      3. Nuridzin DZ, et al
      . Strengthening causes of death identification through community-based verbal autopsy during the COVID-19 pandemic. BMC Public Health 2022;22:1607. doi:10.1186/s12889-022-14014-x
    17. ↵
      1. Sankoh O,
      2. Herbst AJ, INDEPTH
      . Comprehensive health and Epidemiological surveillance system (CHESS) - an innovative concept for a new generation of population surveillance from the INDEPTH network. Lancet Glob Health 2015.
    18. ↵
      1. Sankoh O, INDEPTH Network
      . CHESS: An innovative concept for a new generation of population surveillance. Lancet Glob Health 2015;3:S2214-109X(15)00180-1. doi:10.1016/S2214-109X(15)00180-1
    19. ↵
      1. National Statistics Office
      . Census summary figures national statistics office, 2011. Available: https://www.nso.gov.pg/statistics/population/
    20. ↵
      1. Pham NB,
      2. Seri M,
      3. Lydia K
      . Mortality data of the comprehensive health and epidemiological surveillance system 2022.
    21. ↵
      1. Pham BN,
      2. Jorry R,
      3. Silas VD, et al
      . Leading causes of deaths in the mortality transition in Papua New Guinea: Evidence from the comprehensive health and Epidemiological surveillance system. Int J Epidemiol 2022:dyac232. doi:10.1093/ije/dyac232
    22. ↵
      1. WHO
      . Verbal autopsy standards: ascertaining and attributing causes of death, 2016. Available: https://www.who.int/healthinfo/statistics/verbalautopsystandards/en/
    23. ↵
      1. Nichols EK,
      2. Byass P,
      3. Chandramohan D, et al
      . The WHO 2016 verbal autopsy instrument: An international standard suitable for automated analysis by Interva, Insilicova, and tariff 2.0. PLoS Med 2018;15:e1002486. doi:10.1371/journal.pmed.1002486
    24. ↵
      1. Jorry R,
      2. Pham NB,
      3. Pomat W
      . Piloting the new who 2016 verbal autopsy tablet based data collection method in Asaro/Goroka PNG medical symposium. Port Moresby 2018.
    25. ↵
      1. Centers for Diseases Control and Prevention
      . International classification of diseases,tenth revision (ICD-10): National center for health Statistics, 2021. Available: https://www.cdc.gov/nchs/icd/icd10.htm
    26. ↵
      1. Murray CJL,
      2. Lopez AD,
      3. Black R, et al
      . Population health metrics research Consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets. Popul Health Metr 2011;9:27.doi:10.1186/1478-7954-9-27
    27. ↵
      1. Serina P,
      2. Riley I,
      3. Stewart A, et al
      . Improving performance of the tariff method for assigning causes of death to verbal Autopsies. BMC Med 2015;13:291. doi:10.1186/s12916-015-0527-9
    28. ↵
      1. Serina P,
      2. Riley I,
      3. Hernandez B, et al
      . What is the optimal recall period for verbal autopsies? validation study based on repeat interviews in three populations. Popul Health Metr 2016;14.doi:10.1186/s12963-016-0105-1
    29. ↵
      1. Landis JR,
      2. Koch GG
      . The measurement of observer agreement for categorical data. Biometrics 1977;33:159.doi:10.2307/2529310
    30. ↵
      1. Trevethan R
      . Sensitivity, specificity, and predictive values: foundations, pliabilities, and pitfalls in research and practice. Front Public Health 2017;5:307. doi:10.3389/fpubh.2017.00307
    31. ↵
      1. United Nations
      . United nations millennium development goals United nations, 2015. Available: https://www.un.org/millenniumgoals/childhealth.shtml
    32. ↵
      1. World Health Organization
      . Sustainable development goals: Health and health related targets, 2015. Available: http://www.who.int/gho/publications/world_health_statistics/2016/EN_WHS2016_Chapter6.pdf
    33. ↵
      1. National Department of Health, World Health Organization
      . Papua new guinea - WHO country cooperation strategy 2016–2020, 2016.
    34. ↵
      1. Mola G,
      2. Kirby B
      . Discrepancies between national maternal mortality data and international estimates: The experience of Papua New Guinea. Reprod Health Matters 2013;21:191–202. doi:10.1016/S0968-8080(13)42725-8
      OpenUrlPubMed
    35. ↵
      1. Sanga K,
      2. de Costa C,
      3. Mola G
      . A review of maternal deaths at Goroka general hospital, Papua New Guinea 2005–2008. Aust N Z J Obstet Gynaecol 2010;50:21–4. doi:10.1111/j.1479-828X.2009.01116.x
      OpenUrlPubMed
    36. ↵
      1. Joint Agency Task Force
      . Media statement: PNG COVID-19 national pandemic response. Port Moresby: Office of the Controller 2021.
    37. ↵
      1. Centers for Diseases Control and Prevention
      . Symptoms of coronavirus, 2021. Available: https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html
    38. ↵
      1. Pham BN,
      2. Jorry R,
      3. Maraga S
      . Chess September 2021 technical report: household socioeconomic status and implication for national response to COVID-19 2021.
    39. ↵
      1. WHO
      . Verbal autopsy standards: ascertaining and attributing causes of death tool, 2022. Available: https://www.who.int/standards/classifications/other-classifications/verbal-autopsy-standards-ascertaining-and-attributing-causes-of-death-tool

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors BNP designed CHESS; conceptualised the paper; analysed and interpreted the data; and drafted, revised, finalised and submitted the manuscript. NA, SM and RJ supervised the fieldwork, collected and analysed the data, and provided the inputs. GSJ, VDS and TA collected and analysed the data. TO reviewed and provided feedback for the development of the study. WP oversaw the PNGIMR and approved the submission of the manuscript. All authors have read and approved the manuscript. BNP has full responsbility for the work as guarantor and the conduct of the study, had access to the data and controlled the decision to publish.

    • Funding CHESS was operated with financial support from the PNG government through the Department of National Planning and Monitoring (project no. 23141, PIP no. 02704).

    • Disclaimer The funder had no role in study design, data collection and analysis, or writing of the manuscript.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.