Aim

The aim of the trial is to address the research question: what are the effects of an innovative paper-based HIS (PHISICC) on data use and quality, quality of health and HW perceptions compared with the current HIS, in rural PHC settings?

Patient and public involvement

There was no public or patient involvement in the design of the study or selection of study areas because the intervention being assessed in these trials target healthcare providers and decision-makers, rather than patients or the public in general. Population in the catchment area of selected HFs, potentially using their services, were only approached in order to assess the studies outcomes.

On the other hand, we have involved health systems stakeholders and front-line HWs. Ministries of Health (MOH) at several levels participated in the preparation of the research proposal (personal consultations), in the characterisation of HIS that preceded the trials (countries workshops), and throughout all project components (additional workshops, newsletters and personal communications). Front-line HWs in the three countries have cocreated the intervention (ie, paper-based tools) through workshops, personal feedback and piloting under real live conditions. Some of them are part of the research team and coauthoring this manuscript.

Study design

The study is a CRCT in each of the three countries. In each setting, 70 HFs are randomised to intervention or control (35 per arm). The intervention arm HF use the new PHISICC tools (substituting the usual HIS tools) and the control arm HF use the regular HIS tools. The trial is implemented in the real life contexts of HF carrying out their usual duties.

The trials started between the end of 2019 and beginning of 2020, depending on the country, when the intervention was installed and the baseline surveys carried out. Data collection will last until mid-2021.

Study areas

MOH officials in several countries were contacted before submitting the proposal to the funding agency in order to explore the willingness to engage in a project focusing on paper-based tools. Officials in several countries rejected the offer on the grounds of upcoming digitalisation plans of the HIS in the country. We partnered with MOH that found the research relevant to their context in three countries.

In each country, the eligibility criteria of study areas were that they had to belong to the operational area of research partners; contain a large enough number of HFs and their catchment population; include vulnerable population (eg, with low vaccination coverage, high childhood mortality); and be comparatively neglected in terms of infrastructure and services. We excluded areas with concurrent research or other types of activities that could conflict with the CRCT (such as the coexistence of another health-related study, massive developments in infrastructure or activities involving migration of the population, such as temporary work sites or changes in working sites) and areas with threats to safety or security that could jeopardise research activities.

The study areas are located in Adzopé, Agboville, Tiassalé and Sikensi districts (Côte d’Ivoire); in Funhalouro, Govuro, Homoine, Inhambane, Inharrime, Inhassoro, Mabote, Maxixe and Panda (Inhambane province, Mozambique) and in Yala Local Government Authority (Cross River State, Nigeria).

Eligibility of HFs

The intervention is implemented at the HF level. The eligibility criteria of the HF were that they had to be located in the study areas, belong to the governmental health sector and their main activity should be the delivery of PHC services. HFs were excluded if they had specialised clinical services, inpatients, physicians providing care or with plans for staff turn-over.

A ‘master list’ of eligible HFs was prepared based on information provided by the MOH across all study areas. We aimed at selecting 70 of the eligible HF in each country, using simple random techniques in R.10 See in figure 2 the selection and allocation trial flow chart.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

CONSORT diagram: trial flow chart. CONSORT, Consolidated Standards of Reporting Trials; HF, health facilities.

Allocation and blinding

Allocation of HF into the intervention and control arms took place in a formal event, gathering research partners and MOH officials to offer transparency and promote study ownership by local and national authorities. Equally sized, folded pieces of paper with the names and codes of included HF written on them were introduced in an opaque receptacle where they were manually and blindly mixed. A second receptacle contained two equally sized pieces of paper, one with the word ‘intervention’ and another one with the word ‘control’. A selected person in the meeting, not belonging to the research team, extracted one piece of paper at a time to reach half the number of included HF. Then, a paper was extracted from the second receptacle to assign those HF to the intervention or control arms. The rest of the papers were extracted as well to verify completeness and no duplication of names, and those HF were assigned to the other arm.

Once HFs were selected, all villages or settlements for each health facility catchment area were listed and three in each catchment area were selected. In practice, we selected all villages because the numbers were below (in Côte d’Ivoire) or just above (in Nigeria) the needs. For each village, we used Google satellite maps to identify and geo-locate every visible roof. Where there were many houses per village (roughly, more than fifty or so), a researcher would mark four points in the map slightly beyond the northernmost, southernmost, easternmost and westernmost roofs seen and 30 random points were selected within that square. From the mapped points, 10 per village (with 20 more acting as reserve) were randomly selected and marked on another map used in the field for data collectors to approach households. Where technical problems impeded this approach in a given village, a field supervisor would rotate a bottle on the floor towards the centre of the village and would select at random 10 households in the direction pointed by the bottle, from the outer limit of the village till the centre.11

Blinding is only feasible for the research team members carrying out the CRCT data collection and the analyses of the CRCT findings. The intervention (ie, paper tools) are by design very different from the existing system and it is not possible to blind participants or principal researchers.

We already had the agreement of the MOH and selected HF compliant with the inclusion criteria were provided with the intervention shortly after completing the baseline data collection. Therefore, recruitment as such took place at the same time of the allocation of HF into intervention and control arms.

The intervention

The PHISICC paper-based intervention is a full set of paper-based tools to support decision making by front-line HW. These are the only tools to be used by HW in the intervention arm. The PHISICC tools encompass the whole system (ie, recording and reporting) and all clinical and public healthcare areas and are characterised by a common visual language (eg, spaces for digits and text), and standardised formats across healthcare areas. To support front-line HWs decision making, the PHISICC tools incorporate specific places to explicitly record critical data items (eg, respiratory rate in infants), graphic artefacts to distinguish severity degrees of signs or symptoms (ie, small square for ‘normality’, diamond for ‘attention’ and bold diamond for ‘critical severity’); space to document diagnoses and treatment decisions; and aides memories in the first page of register books.

The PHISICC tools have been developed from May 2017 till June 2019, including production, using a HCD approach.12 A strength of the HCD approach is its ability to unlock the user’s perspective so that designers can build solutions that are fully reality-based and work well. Cocreation groups were formed in each country with researchers, staff from partner institutions and healthcare workers, led by a team of professional designers. Research team members supervised and coordinated exclusively the feedback on the contents of the tools, to ensure compliance with each country clinical guidelines. At the outset of the process, the design focused on three healthcare areas (ie, antenatal care, vaccination and sick child) and slowly extended the new visual language to other healthcare areas. Initial workshops served to brainstorm on problems and potential design solutions, without any other rule than being comprehensive and not rejecting a single idea. Designers, then, formalised some of the most promising solutions and a first round of exchanges within the cocreation team was used to address misinterpretations or inconsistencies. There were two in-the-field testing rounds in Mozambique, two in Côte d’Ivoire and three in Nigeria and uncountable exchanges through teleconferences and email, in-between. The prototypes were considered final when no errors were detected, were compliant with data needs in each country and comments from the field could not be accommodated in the design concept or there was no consensus on minor or formal issues being raised.

The PHISICC tools have been produced in French for Côte d’Ivoire, in Portuguese for Mozambique and in English for Nigeria, which are the official languages used in the health systems in the three countries. They include the official logo of the MOHs. Healthcare areas covered include: family planning, antenatal care, including tetanus toxoid vaccination, delivery, postnatal care, vaccination, sick child, adults outpatient consultation, tuberculosis diagnosis and treatment, and HIV. Referral forms were also designed.

The PHISICC tools have three subcomponents: registers, tallies and reports. Registers are formed by seven Deutsches Institut für Normung (DIN) -A3 and one DIN-A4 (for referrals) book covering all healthcare areas except for tuberculosis treatment, for which DIN-A3 cards where used. Register books have 100, 200 or 400 pages depending on the country and healthcare area. They are used to record individual clients’ data for each healthcare event, either of clinical or public health nature. Some register books have clinical notes at the very beginning, as ‘aide memoires’, and an example of a filled-in form, to assist HW when doubting how to proceed.

Tallies are DIN-A3 single sheets which contain a list of the indicators to be transferred to higher levels of the health system, with a series of small ovals, grouped in fives, to mark with tally sticks with a pen. In contrast to the current systems that have no tallies or only for vaccination, tallies were created for all healthcare areas. In the middle-right side of the tally, a column accommodates cells aligned with the ovals to insert the count for each indicator; and in the far right of the sheet there is a replica of the count column, separated with a perforated line, which is detached and sent, as part of the monthly report to the higher level in the health system.

While current HIS tools are consistently organised in tabular formats and books, where each clinical event is recorded in a row and each variable (eg, age, gender, HIV status, diagnosis) in a column, PHISICC tools incorporated several innovations; in summary: a visual language to guide the clinical decisions of HWs based on severity (ie, if it is recorded that a child has convulsions, a visual artefact indicates severity), more space for clinical data (eg, vital signs), inclusion of all critical information to assess patients (eg, obstetric history, gestational age, fundus height, breath rate in infants), consolidation of information of all antenatal care visits for a single pregnant woman in the same page, among many other formal and contents improvements, including improved aesthetics. A systematic comparison of the new (intervention) and current (control) tools is provided in table 1.

We aimed at creating ‘a system’ (not just some tools) focusing on decision making by front-line HWs. The epidemiological and public health contexts in the three countries are similar, as confirmed by the similarities in the existing HIS between the three. The visual language and the recording forms where common to the three countries because clinical decisions are common to the three; although forms allowed for specific tests or treatments. The reporting component was adapted to each country set of indicators, although the visual language and reporting processes were harmonised.

During 3 or 4 days, HWs were trained on HIS before the start of the trial. In the intervention arm they were trained on the PHISICC tools; and the control arm received a refresher training about the regular tools, during the same number of days.

Additionally, given that the regular tools already contained information on past vaccination history of children still to complete their vaccination schedule, we created a mechanism to retrieve data of children’s vaccination status to transcribe into the new vaccination register book in the intervention arm (‘system transition’).

Tools were endorsed by MOH, printed in local printing companies and distributed to HW at the end of the training sessions. A digital spreadsheet was created to monitor consumption and order additional tools to cover health facility needs during the life of the trial.

Outcomes

There are five primary outcomes (table 2). Vaccination adherence is defined as the total number of vaccine doses given in the correct time interval to children in households in the HFs catchment villages over the total number of vaccine doses that should have been given during the same period. Antenatal care visits uptake will also be considered depending on the expected number of pregnancies in the study areas. Both are used as proxies for health outcomes in terms of protection against disease13 and prevention of pregnancy complications14 and are assessed in a random sample of households in the HFs catchment areas. Data concordance is defined as the level of agreement of HIS indicators between (1) records of healthcare events (recounts), (2) tallies (recounts) and (3) reports (aggregated data to higher levels of the system).3 Decision making will be assessed considering the diagnostics scope in the sick child (ie, number of different diagnoses per sick child consultation) and treatment appropriateness (ie, number of prescribed treatments that are supported by a documented diagnosis). These outcomes will be assessed in a random sample of records and corresponding reports during the last 4 months of the study period. HWs satisfaction will be assessed in all HWs in included HFs using a standardised questionnaire.15–17 While the intervention targets HF, some of the outcomes are measured at the level of HF and some others in the communities of HF catchment areas.

Secondary outcomes are classified under the following domains: data quality, data use, mortality, HW experience, client experience and resource consumption:

• Data quality, assessed in a sample of records

  • Completeness of recording and reporting in specific forms; that is, prevalence of unduly missing data items, partograph used.

  • Accuracy of recorded figures in comparison to real events (eg, physical counting of commodities, such as number of 500 mg paracetamol tablets as recorded versus number of 500 mg paracetamol tablets as counted).

  • Timeliness of reporting, as documented by time stamps in forms.

  • Loss of data or data which does not reach the next upper administrative level.

• Data use, assessed in a sample of records

  • In terms of knowledge (eg, vaccines due based on date of birth; weight for length assessments).

  • Cases of different conditions properly treated (eg, diarrhoea cases given oral rehydration therapy according to national guidelines; pneumonia cases given appropriate antibiotic according to national guidelines).

  • Public health decisions: availability of lost to follow-up lists or plans for vaccination, tuberculosis or HIV/AIDS treatment control.

  • Occurrence of stock outs of essential drugs.

• Overall under-5s mortality and under-5s mortality excluding perinatal mortality,18 in a sample of households in HFs catchment areas.

• HWs’ ‘human experience’ and satisfaction (all HWs).

• District health information officers’ ‘human experience’ (selected healthcare programme managers).

• Clients’ ‘human experience’ and satisfaction, in a sample of households in HFs catchment areas.

• Resources consumption (eg, time use, costs)

  • Intervention costs: tools, training, start-up.

  • Time used for recording and reporting (eg, time-motion study).19

  • Cost-effectiveness per unit of additional improvement in outcomes of interest.

It is worthwhile to note that outcomes that do not relate to data quality and use will be assessed using additional data collection tools (eg, survey questionnaires), which are the same for intervention and control HFs. Hence, the effects of the intervention cannot be attributed to the changes in performance of the paper tools routinely used to record healthcare events in intervention and control HFs, which are different by design.

In addition, we will consider ‘explanatory outcomes’ that will help to understand how the measured effects have taken place and why. We will look at the details of the interplay between the intervention, the system, the users and the context. Process indicators will be based on the documented activities that have taken place, from the conception of the intervention, up to its implementation, monitoring and evaluation. Process indicators may include: intervention setup and implementation, monitoring of the use of the intervention, special activities targeted at vulnerable populations, district reactions related to the intervention, handling of data coming from the new system, sustainability based on costs information and perceptions, alignment with national health policies and donor priorities. We will also explore healthcare services characteristics looking at generic indicators from HFs, such as human resources profiles and relations with the communities, population characteristics and system and context characteristics captured in early stages of the project, where data are available.

Sample size calculations

The required sample sizes for each primary outcome were determined using simulation to incorporate the clustering (table 1) and to take the baseline and end-line surveys into account. Briefly, we simulated 1000 trials with variation between them caused by drawing different samples from the same distributions. We then used the regression models detailed in the data section to analyse each of the simulated trials and estimate the power as the proportion of trials which detected the effect of the intervention as significant. The simulation code was written in R (online supplemental files 2 and 3).

For each country, we require the probability of α, a type I error (rejecting the null hypothesis when it is actually true) to be less 0.05 and the power to be at least 80%.

For vaccination adherence, using a sample size of 35 HF per arm, we would have 80% power in each country to detect as significant a difference between a proportion of due vaccines given from 75% in the control to 85% in the intervention arms, assuming one child per household, 30 households per HF and a between-HF variation equivalent to a k of 0.25, where k is equal to the SD divided by the mean. The value of k is unknown, but was chosen in line with general observations by Hayes and Bennett.20

For data quality outcomes, with 35 HF per arm we would be able to detect as significant a difference from a ratio of 0.7 (reported:recorded) vaccinations in the control arm to 0.8 with the intervention with 80% power, assuming 100 recorded vaccinations per HF and an SD of 0.25 in the ratios between HF.

In terms of diagnostic scope, we would be able to detect an increase in the proportion of child visits with more than one diagnosis from 30% to 35% with 80% power with 35 HF per arm, 60 records per HF and assuming a k of 0.25.

We would be able to detect as significant an increase from 50% of treatments having a corresponding appropriate diagnosis to 60% with 80% power assuming 35 HF per arm, 1 treatment per child, 25 children per HF and variation between HF corresponding to k=0.25.

For the outcome related to HWs’ satisfaction, we would be able to detect as significant an increase from 50% of HWs satisfied to 75%, with 80% power assuming 35 HF, three HWs per HF and a variation between HF equivalent to k=0.25. Since this variable is measured in the end-line survey only, we used the formula in Hayes and Bennett.20

In summary, in each country we require 35 HF per arm, 3 HW per HF, 100 vaccination records per HF, 60 sick child records per health facility and 30 children per health facility catchment area.

Data collection and management

Data collection took place at baseline and will take place again at the end of the study. Data are collected from HFs, from the households in the catchment areas of the included HFs and also from district offices.

For data quality and data use outcomes, HF registers, tallies and reports will be scrutinised. For population-based outcomes, we carry out household surveys at baseline and at end-line. We use standard approaches for these types of surveys.21 Households are visited, the research project is briefly introduced and consent requested. Ideally, mothers of alive children or women in childbearing age is interviewed in order to obtain information on living children (ie, vaccination history) and death events, respectively, using home-based records if available and accessible. Patients’ satisfaction is assessed using the Patient Satisfaction Quaestionnaire (PSQ-18).22 ,23 ,24 Essentially, the tool enables practitioners to investigate the extent to which their healthcare service meets the perceived needs of their client group and pinpoint areas for improvement.24 The interview will be conducted with consenting individuals as close to their care encounter as possible.25 Data tools are translated into the official languages of the study countries and pilot tested for consistent meaning and relevance to the setting. Data collectors are also able to communicate in local languages. The Satisfaction of Employees in Health Care survey is a validated tool to assess staff satisfaction. It was first developed and validated in a low-income country (Ethiopia)26 and later successfully validated in a high-income country (USA).27

We use a mix of paper and electronic data (Open Data Kit (ODK)28) collection tools. Data collectors are trained to minimise error. Tools are piloted before implementing. ODK data are regularly stored and sent to secure servers, as soon as data collectors reach their office base. Data from paper tools are double entered and compared and sent to secure servers. Each data collection tool has its corresponding electronic database that is cleaned and submitted to the analyses. All data are anonymised at the point of data collection or as soon as possible in the data management process. Data are labelled with an arm code (eg, ‘A’ or ‘B’) without any further information allowing to disclose which data items belong to the intervention or to the control arms, ensuring blinding during data analyses.

Quality will be assured through several mechanisms: piloting of data collection tools; thorough training of field workers; checking missing data related; double, independent data entry from papers into digital databases; early descriptive analyses to detect potential outliers; fieldworkers tracking and supervision.

Data analysis

The analysis will be carried out for each country separately, and based on intention to treat.

Baseline population and health facility characteristics (ie, basic demographic characteristics of population and HWs, professional profile of HWs, health facility size and services) will be summarised. If large imbalances are observed at baseline, the variables can be used to adjust the effect estimate comparisons.29 30

The analyses vary for the different primary outcomes due to the unit of measurement and levels of clustering, the type of variable, and whether measurements were taken at baseline and endpoint or endpoint only. We use regression models to allow us to estimate the effect of the outcome while flexibly accounting for these issues and allowing adjustment for potential confounders.

Logistic regression will be used for the binary variables: vaccine adherence is measured by determining whether each vaccine due was received, and treatment appropriateness by whether each treatment was correctly prescribed. Data concordance and diagnostic scope are count variables and may be analysed with Poisson regression, depending on their distribution. The regression model for HW satisfaction will depend on how it is distributed.

The outcomes have different levels of clustering (children or consultations, HW, HF). We will account for these levels of clustering by including random effects in the regression models.

Four of the primary outcomes are measured at baseline and end-line. The effect of the intervention will be estimated using an interaction term between arm and survey in the regression models: that is, is the change in the outcome between baseline and follow-up in the intervention arm different to the change between baseline and follow-up in the control arm? The effect of HW satisfaction in Nigeria, measured only at end-line, will be estimated as the difference between the intervention and control arms.

All estimates for the effect of the intervention will be presented with 95% CIs. The analyses will be carried out using R.31

Measures to minimise bias

Statistical analyses will be carried out blindly, without knowledge of what HFs or population in the catchment area belong to the intervention or control arms. Only when the analysis code is considered as definitive and fixed, will results be shared with the wider investigators team and the arms for HFs and population will be disclosed.

Outcome measurement bias may take place where data from the HIS, which is the focus of the intervention, is used to measure outcomes. However, we will minimise this by assessing population based outcomes at household level.

Contamination (ie, the intervention affects individuals or units assigned to the control arm) may happen via the exchanges between HWs from HFs in both arms; for example, in monthly district data quality meetings, managerial meetings; or through inputs from supervisors who influence control HFs with intervention tips encountered in HFs of the intervention arms. One mechanism to address this issue is using a district-based cluster randomisation scheme. However, we consider that (1) contamination, despite increasing the awareness of health works in control HFs, will hardly influence the decision making mechanisms that the HIS intervention focuses on and (2) randomisation at the level of district poses additional challenges that are not worth the marginal benefit of reducing a doubtful contamination.32

The spill-over effect (ie, benefits of the intervention extend beyond their direct recipients)33 may take place in higher levels of the health systems; for example, district data managers and programme managers may experience the benefits of better structured and more timely data produced in HFs in the intervention arms. The trial will have no capacity to quantitatively account for spill overs at higher levels of the system, due to the limited number of higher level administrative areas that will be involved in the trial. However, through process indicators, we will consider potential benefits and harms of the intervention at higher levels of the system.

A challenge is the Hawthorne effect (ie, observer effect). Both HWs in the intervention and in the control sites will have an awareness of being observed as data collection activities will be at the same level of intensity in the two arms. Therefore, there should be no differential effect.

Analyses will be based on the intention to treat. It is important to closely monitor if the intervention HFs consistently use the new HIS tools and approaches. The data collection team and the trial monitoring team will check if old forms are still being used in the intervention HFs. However, we do not expect HFs to migrate between intervention and control arms, or vice versa, due to feasibility issues. On the other hand, some household members in a given catchment area may decide to seek for healthcare in a health facility belonging to another trial arm. In these cases, households will be analysed as belonging to the original trial arm.