Article Text
PDF
Abstract
Objectives We evaluate and compare manually collected paper records against electronic records for monitoring the weights of children under the age of 5.
Setting Data were collected by 24 community health workers (CHWs) in 2 Rwandan communities, 1 urban and 1 rural.
Participants The same CHWs collected paper and electronic records. Paper data contain weight and age for 320 boys and 380 girls. Electronic data contain weight and age for 922 girls and 886 boys. Electronic data were collected over 9 months; most of the data is cross-sectional, with about 330 children with time-series data. Both data sets are compared with the international standard provided by the WHO growth chart.
Primary and secondary outcome measures The plan was to collect 2000 individual records for the electronic data set—we finally collected 1878 records. Paper data were collected by the same CHWs, but most data were fragmented and hard to read. We transcribed data only from children for whom we were able to obtain the date of birth, to determine the exact age at the time of measurement.
Results Mean absolute error (MAE) and mean absolute percentage error (MAPE) provide a way to quantify the magnitude of the error in using a given model. Comparing a model, log(weight)=a+b log(age), shows that electronic records provide considerable improvements over paper records, with 40% reduction in both performance metrics. Electronic data improve performance over the WHO model by 10% in MAPE and 7% in MAE. Results are statistically significant using the Kolmogorov-Smirnov test at p<0.01.
Conclusions This study demonstrates that using modern electronic tools for health data collection is allowing better tracking of health indicators. We have demonstrated that electronic records facilitate development of a country-specific model that is more accurate than the international standard provided by the WHO growth chart.
- NUTRITION & DIETETICS
- PREVENTIVE MEDICINE
- PUBLIC HEALTH
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Two primary data sets that are evaluated are collected under the researcher's control and satisfy data quality assurance requirements.
The study was able to quantify data quality using standard statistical tests.
Two primary data sets (paper and electronic) cover a relatively short period of time for longitudinal data analysis.
There was no access to the original data used to derive the international standard, hence we had to derive it from the published model.
Introduction
In a fast paced world with an increasing role for information technology, there is a movement to make a transition in data collection from traditional paper-based records to an electronic, technology-enhanced process of data collection. The health sector is one of the areas where the potential advantages of electronic health records (EHRs) are becoming increasingly recognised.1 In particular, if the goal of the health sector is to have evidence-based, adaptive and agile decision-making, real-time data are imperative.2 Data-driven methods for tracking progress are particularly important for monitoring the health of newborns.3 Historically, the main form of health data collection has been paper records. Storing and updating paper records has proved challenging and most developed countries are moving towards an electronic system.1
However, there is little empirical evidence that electronic data is superior in quality.4 In particular, developing countries are struggling with problems such as delays in obtaining relevant data, access to healthcare and human resistance to automation, all of which affect data quality.4 In Rwanda, as in many developing countries, health data are collected by community health workers (CHWs)5 ,6 who are volunteers, and focus mostly on children's health, vaccination and malnutrition. To test a hypothesis that electronic data is better quality than data from a standard paper-based system, a custom mobile application was given to 24 CHWs who collected electronic health data for monitoring children's growth and development, and stored them as EHR. Then paper records from the same CHWs were compared with the electronic recordings. The purpose of this paper is to visually demonstrate and quantify the differences in data quality, and test whether or not electronic has higher usability than paper recording, defined as better-satisfying intended use of the data. In addition, both are compared with the golden standard provided by the WHO growth chart on children's development.
Previous work
In the literature on the comparison between electronic and paper recording systems, the findings are encouraging in that electronic recording improves quality, measured by consistency. For example, comparison between pen-and-paper interviewing (PAPI), and computer-assisted personal interviewing (CAPI), lists two main advantages of computer surveys: the elimination of data entry and consistency checks during interviews.7 The authors identify two different consistency checks: impossible and improbable data. The findings are that CAPI significantly reduces the error of inconsistences. Two-thirds of the errors aimed at detecting violations of the questionnaire's routing scheme occur when answering questions meant to be skipped, or skipping questions meant to be answered.
In contrast, the use of Personal Digital Assistant (PDA) for consistency checks for profit and sale measurements, which are often noisy, finds consistency checks to be rather limited.4 In the cross-section portion of the survey, it uses consistency checks to compare profit with sales, and prompts when the profit is missing, while in the time-series portion it flags large changes from one period to the next. The conclusion is that the vast majority of changes are justified and the volatility genuine, so it does not recommend PDAs on basis of improvement in measurement and data consistency.
In the health field, studies are evaluating the potential of electronic recordings to improve compliance to treatment and data completeness. One study2 tests if electronic technology could improve adherence to the Integrated Management of Childhood Illness. The authors compare it to the current paper-based protocol in Tanzania by randomly selecting 18 clinics, collecting information during the first examination using paper-based protocol, while the next observation is collected electronically. The sample is data from 1221 children, 681 paper and 550 electronic. The authors find considerably better adherence to the protocol with electronic data collection, concluding that the electronic system improves completeness, defined as reduction in missing data.
Another study8 evaluates patient acceptance of an electronic questionnaire to collect breast cancer data in a mammography setting and improve data completeness.8 The authors developed a questionnaire on a tablet computer incorporating prefilled answers and skip patterns, and tested it on 160 women, 74 using paper and 86 using tablet, evaluating data completeness and patient acceptance. They found it feasible (accomplished) and data completeness improved.
Electronic methods for data collection differ, and Wilcox et al9 describe case studies using multiple methods for primary data collection, ranging from paper to next-generation tablet computers. They performed semistructured phone interviews and addressed issues such as workflow, implementation and security of each method, assessing the usability of different tools.
All the studies in the health field found a significant improvement in data completeness, cost reduction, and better adherence to treatment when data is collected and stored electronically. However, none of the studies look at the relative improvement in data quality beyond missing values.
From an application perspective, the health issue we address is significant because Rwanda is battling child malnutrition; research by the World Bank (Bundervoet T. “Malnutrition chapter” by World Bank, Unpublished document, based on The 2010 DHS and the 2012 Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey (CFSVA)) found that 44% of children under 5 years of age in Rwanda suffer from stunting. Stunting is defined as: “height for age below two standard deviations of the WHO Child Growth Standards median”,10 and often reflects the cumulative effects of undernutrition or chronic malnutrition. This measure puts Rwanda on the 11th highest position in the global malnutrition league table.10 In Rwanda, the probability of stunting for children under 5 years of age increases sharply after 6 months of age, peaks at about 20 months and then becomes relatively stable (Bundervoet T. “Malnutrition chapter” by World Bank, Unpublished document, based on The 2010 DHS and the 2012 Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey (CFSVA)). Stunting decreased by a modest 4% over the last decade, a disappointing drop given the large improvements on other development indicators (Bundervoet T. “Malnutrition chapter” by World Bank, Unpublished document, based on The 2010 DHS and the 2012 Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey (CFSVA)).
In order to assess children's growth, the WHO has been developing standardised growth charts since 1970. However, these charts did not adequately represent early childhood growth and, in 2006, the WHO released new growth curves.11
Soon after the newest WHO growth chart was published, Onis et al12 performed a comparison of WHO and Center for Disease Control and Prevention (CDC) growth curves using a visual methodology. They find important differences between them, such as the CDC charts present lower rates of undernourishment and higher rates of overweight than the WHO standards. Another approach was used in a study by Dewey et al,13 looking at the growth of breastfed infants and constructing the growth curve by cubic polynomial interpolation. The authors conclude that, in order to follow WHO recommendations, a new reference data set is needed.
There exists some disagreement about the applicability of the WHO standard to all countries. Vignerova et al14 find that, for Czech children, the WHO growth standard does not apply. They investigated the prevalence of wasting, defined as weight for height below 2 SD of the WHO Child Growth Standards median, using a sample of 34 000 Czech children under 5 years of age. The main difference between their study and the WHO study concerns the sample selection. The Czech study did not restrict the sample to breastfeeding mothers who were non-smokers, whereas the WHO study did. The authors found that many healthy kids would be labelled as suffering from wasting using WHO but not local Czech standards and recommended further evaluation of WHO growth curves before adoption.
In paediatric emergency medicine, estimation of weight in sick children can be performed in a variety of ways, with considerable predictive power. One very popular method is using the Advanced Pediatric Life Support (‘APLS’) formula (weight=(age+4)×2), which is linear and applies to children from birth to 10 years of age.15 Luscombe et al16 propose a different formula (weight(kg) = 3(age)+7) for accurately estimating weight for the 21st century child. Theron's formula17 is the closest to the model in this paper because it is log linear (weight(kg)=e0.17 age (years)+2.197)). The above formulas, unlike the WHO study, estimate a parametric model that could be used with different data sets, and were tested in developing countries.
Data quality dimensions
Defining data quality is the first step in the process of evaluation. Table 1 shows different dimensions of data quality that can be improved by electronic methods of collection. The dimensions are adopted from Baesens’ “Analytics in a Big Data World”.18
Data quality dimensions
The case study provides the basis for considering the following of the above dimensions:
1) Accuracy: The present paper system does not have consistency checks while the EHR approach incorporates those features in the custom mobile application.
2) Consistency: By comparing the values that are input with a model based on previous measurements, it is possible to immediately detect errors and ensure self-consistency by prompting the data collector to revise the measurement.
3) Accessibility: Electronic reporting is timelier than paper reporting since the present paper system is submitted only once a month. The web application designed for the case study displays the data as soon as the network synchronises the mobile app with the web app.
4) Feedback: In a mobile collection system it is possible to get real-time feedback and have a two-way communication channel.
Case study
Access to healthcare is often limited by a shortage of healthcare workers.19 In countries with a low doctor per patient ratio, mobile devices can increase efficiency and improve access to all healthcare providers. Rwanda has an extremely low doctor per patient ratio, estimated, in 2010, as six doctors for 100 000 patients. Rwanda ranks 174th of 193 countries using this measure.20 In comparison, Belgiumi has 449 doctors—and the world leader is Cuba with 591—per 100 000 patients.21 Neighbouring countries are within a closer range, with Uganda having eight doctors and Kenya 14 per 100 000 patients.22
To ameliorate the shortage of health services, the Rwandan Ministry of Health (MoH) created a programme of CHWs in an initiative to improve the population's health status by ensuring access to preventative and curative healthcare services. CHWs are elected by their community and work as volunteers. Their activities focus mostly on children's health, vaccination and malnutrition, as well as community-based activities around hygiene and sanitation. They also report disease epidemics in their area of coverage, and presently there are a total of 40 000 CHWs in Rwanda. The population of Rwanda is 12 million,23 which represents a ratio of 333 CHWs per 100 000 people, getting it closer to the Belgian ratio, albeit CHWs lack the training and skills of doctors.24 In community care, EHR could be a bridge from untrained CHWs to healthcare providers with timely and relevant data.
The case study was performed in cooperation with the Rwandan MoH and Kibagabaga Hospital in Gasabo district. Several approvals were acquired: ethical approval from the Rwandan National Ethics Committee; and data collection approval from the MoH. The liaison with the hospital was a coordinator of CHWs, and 24 participants were randomly selected from CHWs in her jurisdiction. Participating CHWs had education, age and sex typical of the sample of approximately 1500 CHWs in Gasabo district, one of the most populated districts in Rwanda, with approximately half a million people.ii
To test the above hypothesis that CHWs could use smart phones for health data collection, a mobile application was installed on a Tecno P3 smart phone, and given to the randomly selected Rwandan CHWs to monitor children's growth and development, and the evaluation was carried out in urban as well as rural locations, with 12 CHWs in each. The application was designed to be tolerant to delays in the mobile network and optimised for low-resource settings. The primary goals of this study were to assess the quality of health data collected by electronic tools.
CHWs collected electronic data for 9 months on weight gain and mid-upper arm circumference (MUAC)—measures established by UNICEF for detecting malnutrition25—using the smart phone application specifically designed for the study. Rwandan CHWs routinely collect data on those two indicators from children in their village, once a month. For the purpose of analysis we also collected data recorded in paper notebooks from the participating CHWs. The study was carried out from March to December of 2014. By the end of the study, two CHWs, one from each location, dropped out because of equipment loss. Each group of CHWs was separately trained regarding smart phone and mobile application use, and the training was held in community health locations, one in the city of Kigali, and the other in a rural location in Gasabo district.
We also developed a web application for viewing data, and our system provides detailed information and a web interface that can produce summaries and data analytics. The data from our application can be granular and give coordinators the ability to see which child is malnourished in each sector. It also produces summary reports similar to the present paper-based system.
The common issues in an unreliable telecom network have all the characteristics of a delay-tolerant network (DTN). Such a network is concerned with interconnecting highly heterogeneous networks together even if end-to-end connectivity may not be available.26 One reason DTN is applicable in low-resource settings is inadequate infrastructure, which includes intermittent access to the internet. There are many rural places in Rwanda that are not covered by cellular phone networks. In the test areas, cellular network is available but unreliable. The delay tolerant application sends the collected data when it senses a connection to the network, and meanwhile it stores it on the device.
Data description
Along with electronically collected data on children's weight in Rwanda, we used paper records of the same CHWs to assess their quality. Both of these data sets are evaluated against an international standard provided by the WHO model. Table 2 provides a detailed description of the three data sets.
Description of paper, electronic and WHO data sets used for model evaluation
The above data sets were analysed using a log–log relationship, and table 3 describes the models estimated based on these data sets. The model parameters presented are for girls, boys have a different growth pattern but the results for their models are almost the same as for the girls.
Training and testing data sets, model parameter estimates and goodness of fit measured by cross-validation for Rwandan girls
For the weight measurements in paper and electronic recording methods, CHWs were using the hanging scales pictured in figure 1. When using such a scale, a community worker places a child in a sack connected to a scale that hangs in a fixed place. Measurement errors generated because of an imprecise scale or operator oversight is the same in paper and electronic recording methods.
Hanging scales used by community health workers for child weight measurement.
Strengths and limitations
We were able to identify the following limitations of the study:
We had no access to the original WHO data used to derive the international standard, instead we derived it from the WHO growth model presented by the median and SD.
Two primary data sets (paper and electronic) cover a relatively short period of time (9 months) for longitudinal data analysis; hence we had to limit the study to the cross-sectional approach. A longitudinal study would provide a better source for evaluating the predictive power of the model.
We consider the following to be strengths of the study:
Two primary data sets (paper and electronic) were collected under researcher's control; these satisfy data quality assurance requirements.
The study quantifies data quality between different methodologies for data collection and applies a variety of statistical tests.
Methodology
The first step is to compare the quality of the electronic and paper recording approaches. Along with visual inspection of the data, statistical analyses were also used to quantify the quality. We compared a number of parsimonious parametric models for describing the relationship between weight and age, and found that the most appropriate model structure was of the form, log(weight)=a+b log(age). The simplest assessment of the performance of a quantitative model with respect to a particular data set is based on evaluation metrics such as the mean squared error (MSE). In many applications, it is preferable to provide a normalised metric such as the coefficient of determination (R2), which is equivalent to the square of the correlation coefficient.
The second step is to establish the most appropriate model for describing the data sets collected. An international standard is provided by the WHO in the form of its growth chart for children's development. As the WHO growth charts are based on data from other countries, we wish to understand how applicable the WHO model is for children from Rwanda. In order to provide an accurate assessment of the performance of different models and different data sets, we need to avoid overfitting problems. Unfortunately, if the same data are used for estimating the parameters of the model and also for evaluating that model, it is possible that the procedure will over-fit the data and provide overly optimistic results as far as new data are concerned.
To remedy overfitting we used cross-validation, which is a model evaluation method that ensures that the evaluation is based on different data to that used to construct the model. Cross-validation techniques tend to focus on randomly selected subsections of entire data set.27 A k-fold cross-validation approach was used to calculate the performance in each case.
Another test that we applied is the Kolmogorov-Smirnov (K-S) statistic, which quantifies a distance between the empirical distribution functions of two samples. For the two-sample case the null distribution of this statistic is calculated under the hypothesis that the samples are drawn from the same distribution. The distributions considered under the null hypothesis are continuous distributions but are otherwise unrestricted, and the K-S test is non-parametric.
A log–log model was found to be most appropriate after comparing results with a log-linear model such as Theron's formula.17 One explanation might be that paediatric emergency medicine models are used for an approximation and are applied to the children up to the 10 years of age, whereas our model is designed for children with ages ranging from birth to 5 years.
We also employed a simple non-parametric approach to estimate the mean absolute error (MAE) for paper and electronic recordings. For each 3-month period, we calculated the average and used this to quantify the MAE. We use these results to evaluate the differences between electronic and paper MAE along the growth chart.
Results
Visualisation of the plots obtained from the electronic and paper recording methods gives us an idea of the overall quality of the recording process, and of the nature of the growth and its variability with respect to the age of the child. Figure 2 shows the weight versus age of girls in Rwanda using electronic (left) and paper (right). Figure 3 does the same for Rwandan boys, since boys follow a different growth pattern from girls, and all growth charts are separated by the sex of the child.
Electronic and paper data for Rwanda girls.
Electronic and paper data for Rwanda boys.
Table 3 compares four different evaluation studies, based on different data sets for training and testing. In each case, we use the log–log model for describing the relationship between weight and age. We provide the parameter estimates and the cross-validated goodness of fit measure R2.
On the basis of a model structure of the form, log(weight)=a+b log(age), the training data was used to estimate the model parameters a and b. The four evaluation studies listed in table 3 rely on three models, as can be seen from the parameter values. In the following, these three models will be referred to as the Rwanda paper model (Evaluation A), the Rwanda Electronic model (Evaluation B) and the WHO model (evaluations C and D). It was also possible to calculate the goodness of fit using the coefficient of determination, R2, with a cross-validation approach. The R2 values provided in table 4 give quantitative support to the argument that electronic data collection is superior to paper-based data collection for girls and for boys in Rwanda.
R2 values for the model—log(weight)=a+b log(age), for boys and girl using electronic and paper collection
A direct comparison of traditional forecast evaluation criteria, such as MAE and mean absolute percentage error (MAPE), provides a way to quantify the magnitude of the error in using a given model to make a prediction. Comparing a model of the form, log(weight)=a+b log(age) produces the following results for paper and electronic records, with the errors evaluated in units of kilograms. Electronic records provide considerable improvements over paper records, with at least a 40% reduction in error in both performance metrics, as shown in table 5.
MAE and MAPE of regression model evaluated using paper and electronic data sets for Rwandan girls
In order to compare the data collected by the CHWs in Rwanda with WHO growth chart, we plot the growth relationships arising from all three models on the same graph in figure 4. As can be seen, there exists significant disparity. One possible explanation for the disparity between the WHO growth chart and the data collected in Rwanda is that very young children in Rwanda usually benefit from breast feeding up to their first birthday and therefore grow larger than the world average suggested by the WHO charts. When these babies start eating regular food, approximately around 6 months of age, they do not receive all the required nutrients and by the time they are 1½ years old their weight falls below the WHO growth curve.
Three growth charts based to three different data sets for Rwandan girls.
Another comparison is made between Rwanda electronic (Evaluation B) and paper model based on electronic records but coefficient estimates from the WHO regression (Evaluation D). Table 6 shows two forecast performance metrics, MAE and MAPE, for the Rwanda electronic (Evaluation B) and the WHO model (Evaluation D), for Rwandan girls.
Performance of Evaluation B and D for Rwandan girls
From table 6 it can be seen that the Rwanda Electronic model is outperforming the WHO model. The gain in performance of the Rwanda Electronic model is 10% for MAPE and 7% for MAE.
We also tested if this improvement in forecast performance is statistically significant. In order to do this, we applied a K-S test to the absolute percentage errors, which confirmed that the superior performance of the Rwanda Electronic model was statistically significant (p<0.01).
Finally, a simple non-parametric approach was used to compare the accuracy of paper and electronic recording methods using MAE. The MAE for the electronic recording does not vary significantly, and ranges between 1 and 2 kg. Alternatively, the MAE for paper recordings varies significantly over the growth chart with a major decrease in MAE taking place after 3 years of age. The MAE for paper recording fluctuates around 7 kg for the first 3 years, only dropping to a comparable level of error for ages 3–5 years, as shown in figure 5.
Mean absolute error (MAE) for paper and electronic method.
Conclusions
In community care, EHR could be a bridge from untrained CHWs to healthcare providers with timely and relevant data. The case study answers some of the questions posed about the data quality dimensions presented in table 1.
Accuracy: Table 5 presents performance metrics on the accuracy of models based on electronic and paper data. The electronic data improve on the paper data by more than 40% in the case of both metrics. Table 6 has same indicators for Electronic data versus the WHO model for girls. The Electronic model is 10% more accurate than the WHO model.
Consistency: Real-time consistency checks are available for electronic data, paper data has no checks until entered in the spread sheet or analysed at later times.
Accessibility: Electronic data can be accessed on the web application the same day they were recorded. Reports can be customised for a particular purpose.
Feedback: Not implemented in the pilot study but participants expressed interest in real time feedback and two-way communication.
General conclusions from the above case study can be summarised as follows:
Electronic data records for Rwandan children have a goodness of fit, measured by R2, which is more than twice that of the paper data records for both boys and girls.
Electronic data and paper data collected from Rwandan children differ from WHO growth curves, raising doubts about applicability of WHO growth charts to developing countries with considerable malnutrition.
Comparisons of the electronic and paper recording methods with the standard WHO model show that electronic data are closer to the WHO model.
Electronic data improve performance over the WHO model by 10% in MAPE and 7% in MAE. Results are statistically significant using the K-S test at p<0.01.
It is possible that, in this case, the use of new technology has a strong novelty effect with initial adoption, which will wear out as the tools and devices become more commonplace. This enthusiasm when using novel technology has been discussed primarily for introduction of new, technology-enhanced tools in education, from teachers28 and students.29 However, the 9-month period for the case study is likely to be sufficient for such enthusiasm to fade.
This study demonstrates that using modern electronic tools for health data collection that is automatically fed into EHRs, allows better tracking of health indicators. In this paper, we show that electronic records are superior to paper records. We have also demonstrated that electronic records facilitate development of a country-specific model that is more accurate than the international standard provided by the WHO growth chart. The electronic collection of data and development of country-specific growth charts allows real-time monitoring and is likely to become increasingly important as climate change threatens food security and makes the world’s staple food crops less nutritious.
Acknowledgments
Publication of this article was funded by the University of Colorado Boulder Libraries Open Access Fund.
References
Footnotes
Contributors SB designed the case study, supervised data collection and performed data analysis. PM selected the statistical methodology, and chose the model and criteria for the data comparison.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Obtained.
Ethics approval Rwandan Ethics Board, Carnegie Mellon IRB, University of Colorado IRB.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The primary data sets can be obtained (paper and electronic) by writing to the corresponding author at suzana.brown@sunykorea.ac.kr
↵i Rwanda was a Belgian colony until 1962.
↵ii http://statistics.gov.rw/publications/2012-population-and-housing-census-provisional-results