Article Text
PDF
Abstract
Objectives To assess availability and completeness of data collected before and after a data quality audit (DQA) in voluntary medical male circumcision (VMMC) sites in Zimbabwe to determine the effect of this process on data quality.
Setting 4 of 10 VMMC sites in Zimbabwe that received a DQA in February, 2015 selected by convenience sampling.
Participants Retrospective reviews of all client intake forms (CIFs) from November, 2014 and May, 2015. A total of 1400 CIFs were included from those 2 months across four sites.
Primary and secondary outcomes Data availability was measured as the percentage of VMMC clients whose CIF was on file at each site. A data evaluation tool measured the completeness of 34 key CIF variables. A comparison of pre-DQA and post-DQA results was conducted using χ2 and t-tests.
Results After the DQA, high record availability of over 98% was maintained by sites 3 and 4. For sites 1 and 2, record availability increased by 8.0% (p=0.001) and 9.7% (p=0.02), respectively. After the DQA, sites 1, 2 and 3 improved significantly in data completeness across 34 key indicators, increasing by 8.6% (p<0.001), 2.7% (p=0.003) and 3.8% (p<0.001), respectively. For site 4, CIF data completeness decreased by 1.7% (p<0.01) after the DQA.
Conclusions Our findings suggest that CIF data availability and completeness generally improved after the DQA. However, gaps in documentation of vital signs and adverse events signal areas for improvement. Additional emphasis on data completeness would help support high-quality programme implementation and availability of reliable data for decision-making.
- Data quality
- Zimbabwe
- Monitoring and Evaluation
- voluntary medical male circumcision
- operations research
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
We assess data quality in terms of completeness and availability of routine programme data in a voluntary medical male circumcision (VMMC) programme, improving the evidence base for programme improvement.
We find that data quality audits (DQAs), in general, can help increase data completeness on key factors that ensure patient safety; however, gaps identified in data completeness could negatively impact patient care.
We demonstrate the value of routine DQAs to help increase the quality of programme data, adding to the global literature emphasising monitoring and evaluation.
The study was restricted to four convenience sites with available data due to limited time and financial constraints, reducing generalisability.
No qualitative study was implemented in parallel, reducing understanding of the barriers to data quality only at included sites.
Introduction
Accurate, timely and reliable public health data are essential for the delivery of high-quality healthcare services and for ongoing improvement of public health programmes. Though a large amount of effort has gone into developing tools that can assess the quality of public health data,1–3 few quantitative studies report on the effects of these efforts to motivate improvement in data quality in low-resource settings.4 ,5 Data quality remains weak in many low-resource, public health settings.6 We posit that careful review of programme data and interactive discussion of data weaknesses with programme implementers could help motivate positive changes in provider behaviours and programme documentation.
ZAZIC, a consortium founded in 2013 as a collaboration between International Training and Education Center for Health (I-TECH), Zimbabwe Association of Church-related Hospitals (ZACH) and Zimbabwe Community Health Intervention Research Project (ZICHIRE), cooperates with the Zimbabwe's Ministry of Health and Childcare (MOHCC) to implement an integrated voluntary medical male circumcision (VMMC) programme in 21 districts. Ensuring quality service provision is a critical component of ZAZIC's VMMC programme. From March, 2013 through July, 2015, ZAZIC conducted 83 706 male circumcisions (MCs).
ZAZIC implements routine monitoring and evaluation (M&E) activities to identify and address weaknesses in data collection, data reporting and data use. Previous research to measure implementation of, and improvements in, elements of VMMC surgical efficiency in four sub-Saharan African countries demonstrated that these types of M&E efforts may spur positive policy and programme changes.7 As a part of ZAZIC M&E efforts, periodic data quality assessments (DQAs) were performed to assess data availability and completeness. Through one-day site visits, aggregate data that was reported to the MOHCC was compared by ZAZIC to source document data at VMMC sites, identifying discrepancies and weaknesses to be addressed at the site level. Findings were discussed with facility staff at the end of the site visit, and a detailed report with action items was provided to facilities a few weeks later. Any items identified through the DQA were followed-up with phone calls, and if necessary, with an additional site visit. In February, 2015, I-TECH completed DQAs at 10 sites to assess data quality. DQA report findings were quickly shared with sites and follow-up visits were completed for sites where data quality was initially unsatisfactory.
We conducted a retrospective, cross-sectional, quantitative study in a subset of four sites. Our objective was to assess if DQAs were associated with improvements in data quality, focusing on data availability and completeness. Using concepts from organisational change theory, we posited that our DQA efforts could encourage adoption, implementation and sustainability of facility-level improvements in data quality.8 As our activity was action-oriented and included diagnosis (record review), action planning (feedback provided), intervention (follow-up when needed) and evaluation (ongoing DQAs),9 we hypothesised that targeted post-DQA feedback would lead to significant improvements in both measures of data quality.
Methods
Study design
In February, 2015, ZAZIC conducted DQAs at 10 of the 36 sites participating in ZAZIC's VMMC programme. Several factors influenced a site's likelihood of being selected for the DQA including geographical location and prioritisation of sites providing VMMC to a larger number of clients. Among the 10 sites that received a DQA, four were selected for this follow-up study with preference given to sites found to have poor data completeness during the previous DQA and geographical proximity (figure 1). The focus of this follow-up study was data availability and data completeness of the client intake form (CIF) (see online supplementary file 1). The CIF is the primary tool used by VMMC service providers to document VMMC procedures. CIFs measure seven dimensions of VMMC service provision: demographics, vital signs, preprocedure indicators, eligibility, procedure details, adverse events (AEs) and postprocedure follow-up.
Study flow chart. CIFs, client intake forms; DQA, data quality audit; VMMC, voluntary medical male circumcision.
supplementary file
This cross-sectional, quantitative comparison of CIF data before and after the DQA was conducted using a ZAZIC-developed evaluation tool (described below). CIF data completeness three months before the DQA was compared with results three months after the DQA. CIFs from November, 2014 and May, 2015 were selected to represent the period before and after the DQA, respectively. Selected CIFs were manually reviewed in August, 2015.
Data completeness evaluation tool
With the input of ZAZIC clinicians, 34 CIF variables were identified as the most important indicators of procedure quality (table 1). A completeness score was developed, with one point awarded for each of the 34 CIF variables that had a value recorded.
ZAZIC DQA evaluation tool: list of key variables and their classification
Data collection and analysis
Monthly return forms (MRF), a MOHCC form containing aggregated data on monthly VMMC programme outputs for each site, were used to determine the expected number of CIFs from each site in the selected months. CIFs are organised in binders sorted by sequential client ID numbers to identify missing CIFs. Monthly record availability for each site was calculated by comparing the number of CIFs located to the number reported in MRFs. CIF data was extracted from paper forms, entered using Microsoft Excel and analysed using Stata version 12.0 (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP). Site-specific improvements in CIF availability pre-DQA and post-DQA were compared using χ2 tests, with t-tests used to assess changes in mean data completeness score pre-DQA and post-DQA.
Ethics statement
The Medical Research Council of Zimbabwe, the US Centers for Disease Control and Prevention (CDC) and University of Washington's Internal Review Board provided non-research determination for this routine programme evaluation activity. All patient-level data was de-identified. MRFs and CIFs are the property of the MOHCC and are stored by implementing partners in accordance with MOHCC standards for the routine care of data.
Results
Study population
According to MRFs, a total of 1461 MCs were conducted in November, 2014 and May, 2015 at the four sites. Among the 1400 (96%) clients whose CIFs were located, the median age was 17 years (IQR: 18, 20) ranging from 10 to 47 years old.
Record availability
Before the DQA, the availability of CIFs at sites 3 and 4 approached 100% while at sites 1 and 2, 92.0% and 79.8% of CIFs were available, respectively (table 2). After the DQA, high CIF availability was maintained by sites 3 and 4. For sites 1 and 2, record availability increased by 8.0% (p=0.001) and 9.7% (p=0.02), respectively. This is important because a missing CIF can indicate either failure to complete a CIF for a client or failure to appropriately store the form.
Comparison of pre-DQA and post-DQA client intake form availability by site
Overall changes of data completeness
Results showed that sites achieved high overall data completeness after the DQA visit (table 3). Three of the four sites showed statistical improvement of data completeness on CIFs with the mean percentage of improvement of 8.6% (p<0.001), 2.7% (p=0.003) and 3.8% (p=0.001) in sites 1, 2 and 3, respectively. For site 4, the completeness of CIFs decreased by 1.7% (p<0.001).
Comparison of overall pre-DQA and post-DQA data completeness by site
Changes in data completeness across key variables
Sites showed interesting variations in completeness on seven domains examined in more detail (table 4). Specifically, the eligibility section generally achieved 80% completeness after the DQA; however, completeness of site 3 on this section decreased by 16.8% after DQA. Although the preprocedure assessment section was more complete than other categories at the two time points, any history of bleeding (yes/no) had the lowest completeness among the preprocedure questions. The overall completeness of vital signs records in sites 1, 2 and 3 were over 98%; however, blood pressure (BP) records of site 4 were quite poor at both time points and decreased after the DQA (49.3% in November, 2014 and 8.4% in May, 2015). For site 4, this weak component contributed greatly to a low composite score for vital signs. Overall, in post-DQA CIFs, 97.2% of all incomplete fields on postprocedure records were due to lack of postprocedure BP records. Finally, 7.7% of all CIFs had incomplete AE information. Sites 1 and 2 records were nearly complete at both time points; sites 3 and 4 improved after DQA with improvements of 27.8% and 12.8%, respectively.
The percentage of data completeness by domains of VMMC service provision
Discussion
Our findings suggest that the DQA had a positive effect on data completeness: three of four selected sites demonstrated significant improvements in data completeness after the DQA. In addition, improved AE records and eligibility records suggest that site staff responded to feedback from the DQA, increasing their attention to data quality. In addition to assessing the effects of the DQA, this study also identified remaining gaps that merit attention for continued quality service provision. These persistent gaps remaining after the DQA may be caused by programme–level factors. For example, at site 4 we observed that many individuals had no BP recorded. When we shared this observation with facility staff, we learnt that the facility lacked adolescent BP cuffs, which was a major cause of the poor data completeness. This also highlights an example of how the DQA was able to identify an area where ZAZIC could take action to improve the quality of the VMMC programme. By procuring adolescent BP cuffs following the DQA, ZAZIC was able to improve VMMC programme quality and safety.
Although the DQA improved data completeness, other potential challenges were identified. During the CIF review, we observed that among 395 CIFs from one site, 67.0% and 64.6% of clients had identical preprocedure and postprocedure BP values recorded in November, 2014 and May, 2015, respectively (data not shown). These findings raise serious concerns about the validity of this data point as it is implausible that these patients would have constant measurements before and after the VMMC procedure. BP may not have been measured accurately, or may have been measured a single time with the information recorded as both the preprocedure and postprocedure BP. This issue may be present in other clinics and requires additional follow-up.
Furthermore, the most important objective of the preprocedure assessment is to identify contraindications and prevent AEs; however, we observed that MC contraindications are poorly ascertained on CIFs. Of the contraindications for VMMC,10 some can be identified on physical examination, such as anatomical abnormality and genital ulcer disease. However, other serious contraindications such as bleeding disorders cannot be identified without taking a careful medical history. Among all of the preprocedure assessments listed in the 1400 examined CIFs, 128 were blank and no individuals were documented as having a history of bleeding disorders. As an expected 6.6 per 100 000 men suffer from haemophilia worldwide,11 theoretically, about five cases of haemophilia would be expected in our routine VMMC practice out of 83 706 MCs conducted. Although we would not be expected to encounter a haemophiliac among the 1400 study records we reviewed for this manuscript, failure to completely and properly evaluate contraindications could add more procedure risk within the programme overall. Attention to patient's history must remain a priority. DQAs provide an opportunity for ZAZIC to highlight the importance of these questions through interactive discussions with clinicians at each facility. It is hoped that these action-oriented discussions could ultimately result in improvements in patient care.
Our analysis has several limitations. First, due to limited time and human resources, the study was restricted to four convenience sites with available data. Thus, conclusions may not be generalisable to other VMMC sites. Second, we were unable to explore the barriers that gave rise to low data completeness by interviewing relevant personnel, reducing our ability to translate these findings to practice. Additionally, we are making an assumption that the information recorded on the CIFs is accurate and that improvements in data completeness will correlate with improvements in clinical care and accurate data for decision-making. Finally, we did not assess other dimensions of data quality. For example, it was not possible for us to systematically evaluate the accuracy of data recorded on the CIFs, since we had no source of ‘true’ data for comparison. However, we believe that the improvements we documented in data completeness represent increased efforts by facility staff to correctly document information on CIFs. We believe that this effort will translate to improvements in data accuracy.
Conclusion
Following a DQA, record availability was either maintained or improved in four sites selected for examination of data completeness. Moreover, CIF data completeness improved in three of four sites. We identified several gaps remaining in CIF completion and recommend evaluation of data accuracy and authenticity as part of follow-up efforts. By integrating these assessments into routine practice, CIF data quality will likely continue to improve, providing quality data to inform VMMC programme decision-making in Zimbabwe. Moreover, by consistently providing feedback to those providing and documenting VMMC service delivery, DQAs may lead to sustained improvements in data quality and, ultimately, in the quality of VMMC care.
Acknowledgments
The authors thank the following: The UZ-UCSF Collaborative Research Programme, Zimbabwe Association of Church-related Hospitals (ZACH), Zimbabwe Community Health Intervention Research Project (ZICHIRE) and the China Medical Board.
Footnotes
Contributors YX conducted data collection and review. AFB, CF and YX drafted the paper. BM, MH, MT, VC and SB direct and manage VMMC programme implementation. XY and AB analysed programme data. SX collaborated on the VMMC programme implementation for the MOHCC. CF advised on the project. All authors contributed to the drafting of the paper, reviewed drafts and approved the final manuscript.
Funding This study was supported by the US President's Emergency Plan for AIDS Relief (PEPFAR) through a cooperative agreement, ‘Scaling Up Voluntary Male Medical Circumcision to Prevent HIV Transmission in Zimbabwe’, 1U2GGH000972, from the US Centers for Disease Control and Prevention (CDC), Division of Global HIV/AIDS (DGHA).
Competing interests None declared.
Ethics approval The Medical Research Council of Zimbabwe, the US Centers for Disease Control and Prevention (CDC) and University of Washington's Internal Review Board provided non-research determination for this routine programme evaluation activity. All patient-level data were de-identified. MRFs and CIFs are the property of the MOHCC and are stored by implementing partners in accordance with MOHCC standards for the routine care of data.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The de-identified Stata data file used for analysis is available for sharing.