Do digital innovations for HIV and sexually transmitted infections work? Results from a systematic review (1996-2017)

Objective Digital innovations with internet/mobile phones offer a potential cost-saving solution for overburdened health systems with high service delivery costs to improve efficiency of HIV/STI (sexually transmitted infections) control initiatives. However, their overall evidence has not yet been appraised. We evaluated the feasibility and impact of all digital innovations for all HIV/STIs. Design Systematic review. Setting/participants All settings/all participants. Intervention We classified digital innovations into (1) mobile health-based (mHealth: SMS (short message service)/phone calls), (2) internet-based mobile and/or electronic health (mHealth/eHealth: social media, avatar-guided computer programs, websites, mobile applications, streamed soap opera videos) and (3) combined innovations (included both SMS/phone calls and internet-based mHealth/eHealth). Primary and secondary outcome measures Feasibility, acceptability, impact. Methods We searched databases MEDLINE via PubMed, Embase, Cochrane CENTRAL and Web of Science, abstracted data, explored heterogeneity, performed a random effects subgroup analysis. Results We reviewed 99 studies, 63 (64%) were from America/Europe, 36 (36%) from Africa/Asia; 79% (79/99) were clinical trials; 84% (83/99) evaluated impact. Of innovations, mHealth based: 70% (69/99); internet based: 21% (21/99); combined: 9% (9/99). All digital innovations were highly accepted (26/31; 84%), and feasible (20/31; 65%). Regarding impacted measures, mHealth-based innovations (SMS) significantly improved antiretroviral therapy (ART) adherence (pooled OR=2.15(95%CI: 1.18 to 3.91)) and clinic attendance rates (pooled OR=1.76(95%CI: 1.28, 2.42)); internet-based innovations improved clinic attendance (6/6), ART adherence (4/4), self-care (1/1), while reducing risk (5/5); combined innovations increased clinic attendance, ART adherence, partner notifications and self-care. Confounding (68%) and selection bias (66%) were observed in observational studies and attrition bias in 31% of clinical trials. Conclusion Digital innovations were acceptable, feasible and generated impact. A trend towards the use of internet-based and combined (internet and mobile) innovations was noted. Large scale-up studies of high quality, with new integrated impact metrics, and cost-effectiveness are needed. Findings will appeal to all stakeholders in the HIV/STI global initiatives space.


Strengths of the review
• A Comprehensive and up-to-date systematic review/meta-analysis.
• All digital innovations for HIV/STIs and all health outcomes were reviewed.
• Cochrane methodology and PRISMA guidelines followed.
• Critique of study quality conducted.
• A subgroup analyses performed when similar outcomes were reported.

Limitations of the review
• Cost-effectiveness data from the high HIV/STIs burden regions (i.e., Sub-Saharan Africa and Southeast Asia) were limited.
• A lack of integrated online impact metrics to evaluate internet-based eHealth innovations.
• Studies with small sample sizes, low power, insufficient follow-up time (e.g. 3 weeks or less) sometimes provided contradictory results when objective and subjective metrics evaluated the same outcome.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  HIV/STI infections remain a public health concern worldwide. A million new HIV/STI infections are acquired every day, and their cumulative disease burden exceeds 500 million infections. [1][2][3][4][5] Regarding HIV, many countries are working to meet the UNAIDS 90-90-90 targets; 6 however, structural and societal barriers such as stigma, low socio-economic status, and geographical isolation, impede access to quality care for marginalized populations that are disproportionately impacted by the HIV/AIDS epidemic. [7][8] A lack of timely testing and poor retention in care impairs efforts to control HIV/STIs. 7 9-10 To improve early testing, linkage and retention in care, health care systems around the world are seeking solutions for population engagement, awareness, and education. Providing efficient care to hard-to-reach populations, while plugging gaps in health care service delivery, is urgently needed. [11][12] The World Bank estimates that globally, 96% of the world's population and 70% of the world's poorest have access to a mobile phone. 13 Of seven billion, two billion (30%) individuals own a smartphone and approximately 50% of mobile phone users access the internet through their phones. 14-15 Technological access has created a portal for social media and other internet-based health interventions. 16 The rapid diffusion of mobile phones and internet technologies are prime drivers of this disruption in health care service delivery, through a phenomenon aptly titled, the creative destruction of medicine. 17 Digital innovations such as electronic health (eHealth), mobile health (mhealth), and combined innovations offer promising solutions to improve health service delivery. Ehealth encompasses non-internet and internet-enabled mHealth as well as other internet-based health interventions. These innovations, together with expanded mobile and internet networks, global connectivity, and affordability, present opportunities to change the future landscape of health care service delivery. In recent years, visionary foundations (Grameen, Bill and Melinda Gates Foundation, UNAIDS, Vodafone) have increased funding and created opportunities for innovative thinking in health, as demonstrated by ninety-five countries which have evaluated digital innovations to date. 11 This is most evident in under-resourced settings where low-cost, sustainable solutions to solve complex global health challenges are much in demand. 18 The early use of digital innovations was evident in non-communicable diseases, which gained popularity in infectious disease. 19 In the field of HIV/STIs, a study published in the Lancet was the first to demonstrate the effectiveness of mHealth-based SMS innovations on adherence to antiretroviral therapy (ART). 20 As novel digital innovations, strategies and programs continue to be developed and tested, many smaller reviews and systematic reviews were published. However, a vast majority only evaluated a single innovation (e.g. SMS, social media), focused on one or two outcomes, and restricted their populations to select groups (people living with HIV (PLHIV), pregnant women, adolescents, men who have sex with men (MSM)). [21][22][23][24][25][26][27] These reviews failed to provide a comprehensive summary of all innovations in one place.
Due to the rapid expansion of the field of digital innovations, and with the increased popularity of combined innovations in recent years (beginning 2013), a need for a comprehensive up-todate synthesis on all innovations for HIV/STIs was felt. Our objective was therefore to generate a high quality overview/systematic review, following Cochrane methods and guidelines, to 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  summarize all digital innovations across all populations and outcomes. This review compiles and evaluates all existing data, tailored to inform researchers, policy makers and key stakeholders in the field of HIV/STI on decisions regarding implementation and scale-up. 11 METHODS PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines were followed for the review. 28

Data Sources and Searches
We

Study Selection
Two reviewers independently screened and evaluated citations for eligibility (JD & RV) and two others (BL & SD) independently assessed quality. A senior reviewer was consulted (NPP) for discordance.

Eligibility Criteria
Any clinical trials or observational study designs that evaluated any digital (e/mHealth) technology with any reported outcomes (Refer Figure 1) were included.

Data Abstraction
Two reviewers (RV, JD) independently abstracted all the data. A pre-piloted data abstraction form, was used to abstract the following items: study design, study population, sample size, digital innovation type, HIV/STIs, outcome measures (e.g. impact, acceptability and feasibility), and metrics (e.g. attendance rate, completion rate, satisfaction) (Refer to Appendix 2). We referred to a previously published framework to define and further classify the following metrics for impact, acceptability, and feasibility. 29

Subgroup Pooled Analyses
We classified study designs and then classified digital innovations into three groups: 30 a) mHealth (SMS and phone calls only; i.e. non-internet based); b) Internet-enabled mHealth and other internet-based eHealth (mobile application, website, online campaign, streamed soap opera videos, avatar-guided computer programs); c) Combined innovations (innovations that combined both mHealth (SMS/phone calls) with internet enabled m/eHealth).

Innovations
Digital innovations were documented across the spectrum.

Internet-based m/eHealth:
Studies evaluating internet-based eHealth innovations (21/99) reported results that were largely in favor of the following innovations: social media-based interventions for clinic attendance; avatar-guided and mobile applications for ART adherence; social media, avatar, and soap opera videos for risk reduction behaviors; mobile app for self-care.

BMJ Open
While mHealth (SMS/phone calls only) innovations were highly effective in improving clinic attendance, ART adherence, and turnaround time from testing to treatment, they did not report on other outcomes. It should be noted that SMS and phone calls alone failed to reduce risky sexual behaviors, which was not surprising as it is challenging to reduce risky behaviors with a prescriptive SMS alone. Population engagement is essential for risk reduction through qualitative research.
While internet-based m/eHealth innovations (social media, avatar-guided computer programs, mobile apps, and soap opera videos) demonstrated positive evidence on impact metrics, not all studies reached statistical significance. Those that failed to report a statistically significant improvement in ART adherence had small sample sizes and were underpowered to detect these outcomes (n=76 vs. n=240), and had less frequent sessions over a shorter evaluation period (2 sessions over 6 months vs. 4 sessions over 9 months). 102 107 For mobile applications, studies which reported significant effects recruited participants with varying level of adherence, 104 110 compared with studies which had high adherence at baseline (≥ 95%) and did not show significance (due to smaller changes in effect). For social media-based campaigns, the two studies that did not reach statistical significance in reducing risky sexual behaviors lacked an interactive component and simply displayed educational material, while the study that showed significant effect engaged the participants by allowing them to contact professional cognitive behavioral therapists via live chat sessions. 103 105 117 In terms of quality, confounding and selection bias were noted in observational and quasiexperimental studies, and loss to follow-up in some trials. Nevertheless, the overall validity of the findings from this review was not threatened by biases, as a large proportion of our data was derived from trials. Consistent reporting of metrics was lacking, which prevented a comprehensive meta-analysis. While clinical trials were generally high quality, observational studies were medium to low quality. Objectives, end points, metrics, and measures, are equally important in feasibility studies and must well designed to generate high quality evidence.
Our review is an exhaustive assessment of the role of digital innovations in improving prevention and care for HIV/STIs. Our findings resonate with many smaller systematic reviews, which have separately evaluated individual components of digital innovation, such as SMS-based mHealth. 22-23 130-137 Other systematic reviews evaluating social media-based interventions reported similar findings to ours, in improved testing uptake or improvements in sexual health. 25-27 138-139 Our review makes a valuable addition to the growing body of evidence by highlighting the success of other interactive and engaging innovations such as avatar-guided computer programs, mobile apps, streamed soap opera videos, and combined innovations. These are becoming popular, with their power to engage audiences at many levels. Designing combined innovations offers complementarity with media, methods, platforms, and messaging. This complementarity can encourage participant engagement, and improve prevention and care metrics and measures  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57 58 59 60 F o r p e e r r e v i e w o n l y 11 sustainably over time. This is more challenging when only one innovation (e.g. mHealth SMS/phone calls only) is the sole focus.

Caveats and implications for future research
There are some caveats to consider while designing and evaluating digital innovations. Innovations aiming to reduce risky sexual behaviors need to be interactive and tailored to the setting and population, with a deep understanding of patients' needs and preferences. 137 140-141 Any communication with patients should be customized for timing to avoid uptake fatigue. For example, patients may be more responsive to weekly versus daily SMS ART reminders. 32 142 Future research needs to be focussed on tailoring innovations to the context and population, and program objectives.
Study quality is essential to generating meaningful results. Large and representative samples of the underlying population and sound statistical techniques during data analysis can prevent or address selection bias. Exploring reasons for differential loss to follow-up would inform future studies. Wherever possible, a control group should be included to differentiate Hawthorne effect from the effect of the intervention. 143 Trials and impact designs can prevent or reduce confounding. Following checklists, such as the report recently published by the WHO mHealth Technical Evidence Review Group on reporting of mHealth innovations, is suggested and encouraged. 144 Objective measures (e.g. HIV/STIs diagnosis, VL load) are desired in reporting of quantitative outcomes, over subjective self-reported data (e.g. condom use, self-reported adherence). This could potentially reduce some biases (misclassification/desirability bias/recall bias). Qualitative data are rich and complement the understanding of all the contextual and population needs, and capture the dynamics of sustainability and change. They need to be urgently integrated with quantitative data to provide a holistic picture of innovation.
The quality of digital data requires improvement. Across studies, a lack of integrated online impact metrics in evaluating the success of innovations was evident. With continuously evolving digital media, inventing new ways to evaluate acceptability and feasibility becomes necessary. For example, some studies tracked online metrics via Google analytics. 74 100-101 121-124 Synergy with industry powered metrics could be a new wave to measure success.
To scale up proven innovations, a multi-stakeholder engagement is necessary. For that, data and metrics that appeal to all sections of stakeholders are needed. In addition to improving the quality of randomized controlled trials and quasi-experimental impact studies, qualitative studies, cost effectiveness studies, usability studies, are needed.

Implications for policy and practice
In consonance with other systematic reviews, evidence at-scale was scarce. 138 This limits the projection of the long-term effectiveness of digital innovations. More evidence on scale-up, cost savings and cost-effectiveness from Sub Saharan Africa and Asia is needed. Future investments that incentivize both: the development and evaluation of combined innovations by government and industry alike, and focus on sustainability of digital innovations with public and private partnerships, are urgently needed.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  To control HIV/STIs globally, we need novel and disruptive innovations that will uniquely impact health outcomes across the spectrum of access, engagement, treatment and retention so as to impact health service delivery. On one hand, mHealth (SMS/phone calls only) and internetbased m/eHealth were found acceptable, feasible and offered complementarity in improving prevention and care of HIV/STIs. On the other hand, when combined, they provided customized and contextualized solutions for hard-to-reach populations. Integrating these innovations across various levels of healthcare with clear evaluation, monitoring, and documentation of metrics will help enhance existing health service delivery models to impact health outcomes over time.
10.5% reported being tested for HIV during/after the 12W Int. PB: Self-report. Non-significant increase in HIV knowledge & attitudes / No impact. Acceptability: Selfreport.
Acceptable & useful. Majority shared w/ others and want to get tested in future. ATT treatment: Attendance rate.
65% said SMS had no effect on attendance.
75% had no difficulty in receiving and responding to SMS / Highly feasible.
SMS motivated HIV counseling and testing uptake in 89% / Highly feasible. ART in APs: Selfreport + MEM.

SMS
Lim HIV SMS support group+ inquiries answered by physicians.
Overall satisfaction.
SMS easily kept confidential.

Strengths of the review
• An updated and comprehensive systematic review/meta-analysis of all innovations in HIV/STI.
• Evaluation of study quality with biases, subgroup analyses and sensitivity analyses.
• Evaluation of metrics and measures for objective and subjective data.

Limitations of the review
• Limited data were reported from Sub-Saharan Africa and Southeast Asia (29%, 29/99).
• Limited data on cost effectiveness from high burden settings.
• A lack of integrated online impact metrics to evaluate internet-based eHealth innovations. HIV/STI infections remain a public health concern worldwide -a million new HIV/STI infections are acquired every day, with cumulative disease burden exceeding 500 million infections. [1][2][3][4][5] Regarding HIV, countries are working hard to achieve the new UNAIDS 90-90-90 treatment targets; 6 however, structural and societal barriers such as stigma, low socio-economic status, and geographical isolation, impede access to quality care for marginalized populations who are disproportionately impacted by the HIV/AIDS epidemic. [7][8] Likewise, a lack of timely testing and poor retention in care impairs efforts to control HIV/STIs. 7 9-10 To improve early testing, linkage and retention in care, health care systems globally are seeking solutions to improve population engagement, awareness, and education, and efficient care for their hard-toreach populations. It is imperative to plug gaps in health care service delivery. [11][12] Digital innovations such as electronic health (eHealth), mobile health (mHealth), and combined innovations offer promising solutions to improve health service delivery. eHealth encompasses non-internet and internet-enabled mHealth as well as other internet-based health interventions. These innovations, together with expanded mobile and internet networks, global connectivity, and affordability, present opportunities to change the future landscape of health care service delivery.
The World Bank estimates that globally, 96% of the world's population and 70% of the world's poorest have access to a mobile phone. 13 Of seven billion, two billion (30%) individuals own a smartphone; approximately 50% of mobile phone users access the internet through their phones. 14-15 Technological access has created a portal for social media and other internet-based health interventions. 16 A rapid diffusion of mobile phones and internet technologies are prime drivers of this disruptive phenomenon in health, aptly titled, the creative destruction of medicine. 17 In recent years, visionary foundations (Grameen, Bill and Melinda Gates Foundation, UNAIDS, Vodafone) have, with funding, created opportunities for innovative thinking in health. To date, ninety-five countries have evaluated some digital health innovations. 11 This is most evident in under-resourced settings where low-cost and sustainable solutions are needed to solve complex global health challenges. 18 Digital innovations were first used in non-communicable diseases and later became popular in infectious disease. 19 In the field of HIV/STIs, a Lancet study demonstrated the effectiveness of mHealth-based SMS innovations on adherence to antiretroviral therapy (ART). 20 As novel digital innovations and strategies continue to be developed and tested, many smaller reviews and systematic reviews were published. However, a vast majority of these reviews only evaluated a single innovation (e.g. SMS, social media), one or two outcomes, and restricted exploration in select sub-groups (people living with HIV (PLHIV), pregnant women, adolescents, men who have sex with men (MSM)). [21][22][23][24][25][26][27] These reviews failed to provide a comprehensive summary of all innovations for program planning and research. Due to a rapid expansion of digital innovations, and an increased popularity of combined innovations (2013-), a need for a comprehensive up-to-date synthesis on all innovations for HIV/STIs was felt.
Our primary objective was to generate a high quality overview/systematic review that summarizes all digital innovations across all populations and outcomes in HIV/STIs. Our

Study Selection
Two reviewers independently screened and evaluated citations for eligibility (JD & RV) and two others (BL & SD) independently assessed quality. A senior reviewer was consulted (NPP) for discordance.

Eligibility Criteria
Any clinical trials or observational study designs that evaluated any digital (m/eHealth) technology with any reported outcomes (Refer Figure 1) were included.

Data Abstraction
Two reviewers (RV, JD) independently abstracted all the data. A pre-piloted data abstraction form, was used to abstract the following items: study design, study population, sample size, digital innovation type, HIV/STIs, outcome measures (e.g. impact, acceptability and feasibility), and metrics (e.g. attendance rate, completion rate, satisfaction) (Refer to Appendix 2). We referred to a previously published framework to define and further classify the following metrics for impact, acceptability, and feasibility. 29

Subgroup Pooled Analyses
We classified study designs and then classified digital innovations into three groups: 30 a) mHealth (SMS and phone calls only; i.e. non-internet based); b) Internet-enabled mHealth and other internet-based eHealth (mobile application, website, online campaign, streamed soap opera videos, avatar-guided computer programs); c) Combined innovations (innovations that combined both mHealth (SMS/phone calls) with internet enabled m/eHealth). Only one subgroup reported similar outcomes which could be pooled, SMS and phone calls, for the following outcomes: a) clinic attendance with SMS; and b) ART adherence via Medication Event Monitoring System (MEMS) caps, with SMS. We pooled these outcomes using a random effects subgroup analysis. Given the diversity in the sample populations between studies, we used the Dersimonian and Laird random effects frequentist model, weighted by study sample to calculate a pooled effect. We generated forest plots for visual representation of heterogeneity and pooled odds ratios (OR) with 95% confidence intervals (CI). We performed all statistical analyses using Stata/IC, version 13 (StataCorp, College Station, Texas USA). 31

Narrative Analysis
We narratively described all other data using as follows: Digital innovations were classified into the following groups based on the strength of evidence: high/strong evidence (metrics at 75-100%), moderate evidence (51-74%), and low/weak evidence (50% or less).
Acceptability: Acceptability was defined as the receptivity in using digital innovations.
Feasibility: Feasibility was defined as the perceived convenience in using digital innovations. It was reported with various metrics: completion, retention, response and referral rates.
Impact: Impact was defined as a statistically significant improvement in measured outcomes compared to a comparator group (i.e. control group or baseline observations). The metrics used to evaluated impact were: A) attendance rate, B) ART adherence, C) risk reduction, D) self-care and E) partner notification. Impact measures were evaluated on two criteria: effect size and precision. Effect size was assessed when data on a comparator group was made available. Precision of the effect estimate was assessed whenever reported, as it reflects the variance or spread of results.

Quality Assessment
We assessed study quality for both clinical trials and observational studies. We used the Cochrane Risk of Bias Tool for trials, and Newcastle-Ottawa quality assessment scale for observational studies.

RESULTS
Of 4252 citations identified through our extensive search, 792 were selected for full-text screening, and 99 citations met our inclusion criteria and were included in this review for evidence synthesis (Refer: Figure 1).

Combined innovations:
Studies evaluating combined innovations (9/99) showed success of social media + SMS in increasing clinic attendance and partner notification; interactive websites + SMS in improving ART adherence; and mobile app + SMS in increasing self-care. Among the five impact studies, 80% reported a favorable outcome. An online campaign with SMS services increased CT, GC, and HIV tests uptake by 41%, 91%, and 190%, respectively; 123 an interactive website with SMS reminders improved ART adherence in drug-users (n=20; p=0·02); 121 a social media-based partner notification with SMS increased notified contacts by 144% (63.5% in 2011 from baseline 26% in 2010); 126 and a mobile app with SMS significantly improved self-care performance in HIV-infected individuals compared to baseline (n=19; p=0.002). 129

Quality
Studies were individually evaluated on quality criteria and biases were noted. Across trials, losses to follow-up were reported in 31% of RCTs and 55% of quasi-trials. Additionally, biases (i.e. misclassification, recall bias) were of concern in 58% of the RCTs and 64% of quasi randomized trials (Refer to Appendix 4 & 5).
In observational studies, confounding (68%) and selection bias (66%) were observed. (Refer to Appendix 6). Studies with small sample sizes, low power or insufficient follow-up time (e.g. 3 weeks or less) sometimes provided contradictory results when objective and subjective metrics evaluated the same outcome.

Summary of findings
Overall, digital innovations reported positive effects on key metrics. We noted a strong positive effect of digital innovations on clinic attendance rates (70%; 26/37), ART adherence (69%; 20/29), risk reduction behaviors (67%; 8/12) and self-care (100%; 2/2). SMS/phone calls were not able to reduce risky sexual behaviours; however social-media based interventions, particularly interactive social media, were effective in reducing risky sexual behaviors. Acceptability was found to be high for all innovations. Feasibility estimates also remained high for all innovations, except for social media-based interventions, possibly due to a perceived lack of privacy and confidentiality. Combined innovations may thus offer promise in plugging this feasibility gap, with internet-based innovations compensating for limitations in SMS-only strategies and vice versa.
While mHealth (SMS/phone calls only) innovations were highly effective in improving clinic attendance, ART adherence, and turnaround time from testing to treatment, they did not report on other outcomes. It should be noted that SMS and phone calls alone failed to reduce risky sexual behaviors, which was not surprising as it is challenging to reduce risky behaviors with a prescriptive SMS alone. Population engagement is essential for risk reduction through qualitative research.
While internet-based m/eHealth innovations (social media, avatar-guided computer programs, mobile apps, and soap opera videos) demonstrated positive evidence on impact metrics, not all studies reached statistical significance. Those that failed to report a statistically significant improvement in ART adherence had small sample sizes and were underpowered to detect these outcomes (n=76 vs. n=240), and had less frequent sessions over a shorter evaluation period (2 sessions over 6 months vs. 4 sessions over 9 months). 102 107 For mobile applications, studies which reported significant effects recruited participants with varying level of adherence, 104 110 compared with studies which had high adherence at baseline (≥ 95%) and did not show significance (due to smaller changes in effect). For social media-based campaigns, the two studies that did not reach statistical significance in reducing risky sexual behaviors lacked an interactive component and simply displayed educational material, while the study that showed significant effect engaged the participants by allowing them to contact professional cognitive behavioral therapists via live chat sessions. 103 105 117 In terms of quality, confounding and selection bias were noted in observational and quasiexperimental studies, and loss to follow-up in some trials. Nevertheless, the overall validity of the findings from this review was not threatened by biases, as a large proportion of our data were derived from trials. While clinical trials were generally high quality, observational studies were medium to low quality.
Consistent reporting of metrics was lacking, which prevented a comprehensive meta-analysis. Objectives, end points, metrics, and measures, are equally important in feasibility studies and must well designed to generate high quality evidence.
Our review is an exhaustive assessment of the role of digital innovations in improving prevention and care for HIV/STIs. Our findings resonate with many smaller systematic reviews, which have separately evaluated individual components of digital innovation, such as SMS-based mHealth. 22-23 130-137 Other systematic reviews evaluating social media-based interventions reported similar findings to ours, in improved testing uptake or improvements in sexual health. 25-27 138-139 Our review makes a valuable addition to the growing body of evidence by highlighting the success of other interactive and engaging innovations such as avatar-guided computer programs, mobile apps, streamed soap opera videos, and combined innovations. These integrated innovations and programs are gaining in popularity, because of their power to engage rural and urban audiences at many levels.
Designing combined innovations that complementarity of various media, methods, platforms, and messaging may delivery best results. This complementarity may also encourage participant engagement, to improve prevention and care metrics and measures sustainably over time.
Engagement is challenging when only one innovation (e.g. mHealth SMS/phone calls only) is the sole focus, where boredom is likely.

Caveats and implications for future research
There are some caveats to considering design and evaluation of innovations. Future research needs to be focussed on tailoring innovations to the context and population, and program objectives. Innovations aiming to reduce risky sexual behaviors could be interactive and tailored to the setting and population, with a deep understanding of patients' needs and preferences. 137 140-141 Any communication with patients could be customized for timing to avoid fatigue with its uptake. For example, patients may be more responsive to weekly versus daily SMS ART reminders. 32 142 Study quality is essential to generating meaningful results. Large and representative samples of the underlying population and sound statistical techniques during data analysis or sampling methodology, can minimize selection bias. Exploring reasons for differential losses to follow-up could inform future studies. Wherever possible, a control group should be included to differentiate Hawthorne effect from the effect of the intervention. 143 Trials and impact designs can prevent or reduce confounding. Following checklists, like the one by the WHO mHealth Technical Evidence Review Group on mHealth innovations, is suggested and encouraged. 144 Objective measures (e.g. HIV/STIs diagnosis, VL load) are desired in reporting of quantitative outcomes, over subjective self-reported data (e.g. condom use, self-reported adherence). This could potentially reduce some biases (misclassification biases/ or, desirability/recall biases) that are observed with subjective reporting.
Qualitative data are rich and complement the understanding of all the contextual and population needs, and capture the dynamics of sustainability and change. They need to be integrated with quantitative data to provide a holistic picture of uptake of any digital innovation.
Quality of digital data will merit from an improvement. Across studies, a lack of integrated online impact metrics in evaluating the success of innovations was evident. With continuously evolving digital media, inventing new ways to evaluate acceptability and feasibility becomes necessary. For example, some studies tracked online metrics via Google analytics. 74 100-101 121-124 Synergy with industry powered metrics could be a new wave to measure success of digital innovations.
To scale up proven innovations, a multi-stakeholder engagement is necessary. For that, data and metrics that appeal to all sections of stakeholders will be needed. In addition to improving the quality of randomized controlled trials and quasi-experimental impact studies, qualitative studies, cost effectiveness studies, usability studies, are also needed.

Implications for policy and practice
In consonance with other systematic reviews, evidence at-scale and over time was scarce. 138 This limits the projection of the long-term sustainability and cost effectiveness of digital innovations. More evidence on scale-up, cost savings and cost-effectiveness from Sub Saharan Africa and Asia is needed. Future investments that incentivize both: the development and evaluation of combined innovations by government and industry alike, and focus on sustainability of digital innovations with public and private partnerships, are urgently needed.

CONCLUSION
To control HIV/STIs globally, we need novel and disruptive innovations that will uniquely impact health outcomes across the spectrum of access, engagement, treatment and retention so as to impact health service delivery. On one hand, mHealth (SMS/phone calls only) and internetbased m/eHealth were found acceptable, feasible and offered complementarity in improving prevention and care of HIV/STIs. On the other hand, when combined, they provided customized and contextualized solutions for hard-to-reach populations.
Innovations need to be proven for impact and cost effectiveness, using a combination of clinical trials, quasi-randomized studies, observational studies, qualitative research studies. Integrating these innovations across various levels of healthcare with clear evaluation, monitoring, and documentation of metrics will facilitate their integration within existing health service delivery models so as to efficiently impact health outcomes over time.
Findings from this comprehensive review will be informative to all stakeholders -innovators, researchers, healthcare practitioners, policy makers and funders -worldwide seeking evidence on integrating and funding innovations, to make the entire spectrum of HIV/STI care.

ACKNOWLEDGEMENT
The authors would like to acknowledge Ms. Megan Smallwood for her assistance in editing the manuscript.
NPP: data critiquing, write-up, critique, and overall responsibility of the data.
RV, BL and SD: data synthesis, write-up and critique.
JK, TP and KD: write-up and critique.

Competing Interests:
There are no conflicts of interest

Data Sharing:
No additional data are available. This is a systematic review/syntheses of existing studies, therefore all data are reported in the tables.

Mobile app + SMS
10.5% reported being tested for HIV during/after the 12W Int. PB: Self-report. Non-significant increase in HIV knowledge & attitudes / No impact. Acceptability: Selfreport.
Acceptable & useful. Majority shared w/ others and want to get tested in future.
More likely to get vaccine on time after controlling for insurance and site of care (AOR=1.83(1.23-2.71)) / Effective. ATT FU appointment: Selfreport.

SMS
65% said SMS had no effect on attendance.
75% had no difficulty in receiving and responding to SMS / Highly feasible.
Overall satisfaction.

Strengths of the review
• An updated and comprehensive systematic review/meta-analysis of all innovations in HIV/STI.
• Evaluation of study quality with biases, subgroup analyses and sensitivity analyses.
• Evaluation of metrics and measures for objective and subjective data.

Limitations of the review
• Limited data were reported from Sub-Saharan Africa and Southeast Asia (29%, 29/99).
• Limited data on cost effectiveness from high burden settings.
• A lack of integrated online impact metrics to evaluate internet-based eHealth innovations.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60   F  o  r  p  e  e  r  r  e  v  i  e  w  o  n  l  y  INTRODUCTION HIV/STI infections remain a public health concern worldwide -a million new HIV/STI infections are acquired every day, with cumulative disease burden exceeding 500 million infections. [1][2][3][4][5] Regarding HIV, countries are working hard to achieve the new UNAIDS 90-90-90 treatment targets; 6 however, structural and societal barriers such as stigma, low socio-economic status, and geographical isolation, impede access to quality care for marginalized populations who are disproportionately impacted by the HIV/AIDS epidemic. [7][8] Likewise, a lack of timely testing and poor retention in care impairs efforts to control HIV/STIs. 7 9-10 To improve early testing, linkage and retention in care, health care systems globally are seeking solutions to improve population engagement, awareness, and education, and efficient care for their hard-toreach populations. It is imperative to plug gaps in health care service delivery. [11][12] Digital innovations such as electronic health (eHealth), mobile health (mHealth), and combined innovations offer promising solutions to improve health service delivery. eHealth encompasses non-internet and internet-enabled mHealth as well as other internet-based health interventions. These innovations, together with expanded mobile and internet networks, global connectivity, and affordability, present opportunities to change the future landscape of health care service delivery.
The World Bank estimates that globally, 96% of the world's population and 70% of the world's poorest have access to a mobile phone. 13 Of seven billion, two billion (30%) individuals own a smartphone; approximately 50% of mobile phone users access the internet through their phones. 14-15 Technological access has created a portal for social media and other internet-based health interventions. 16 A rapid diffusion of mobile phones and internet technologies are prime drivers of this disruptive phenomenon in health, aptly titled, the creative destruction of medicine. 17 In recent years, visionary foundations (Grameen, Bill and Melinda Gates Foundation, UNAIDS, Vodafone) have, with funding, created opportunities for innovative thinking in health. To date, ninety-five countries have evaluated some digital health innovations. 11 This is most evident in under-resourced settings where low-cost and sustainable solutions are needed to solve complex global health challenges. 18 Digital innovations were first used in non-communicable diseases and later became popular in infectious disease. 19 In the field of HIV/STIs, a Lancet study demonstrated the effectiveness of mHealth-based SMS innovations on adherence to antiretroviral therapy (ART). 20 As novel digital innovations and strategies continue to be developed and tested, many smaller reviews and systematic reviews were published. However, a vast majority of these reviews only evaluated a single innovation (e.g. SMS, social media), one or two outcomes, and restricted exploration in select sub-groups (people living with HIV (PLHIV), pregnant women, adolescents, men who have sex with men (MSM)). [21][22][23][24][25][26][27] These reviews failed to provide a comprehensive summary of all innovations for program planning and research. Due to a rapid expansion of digital innovations, and an increased popularity of combined innovations (2013-), a need for a comprehensive up-to-date synthesis on all innovations for HIV/STIs was felt.

Study Selection
Two reviewers independently screened and evaluated citations for eligibility (JD & RV) and two others (BL & SD) independently assessed quality. A senior reviewer was consulted (NPP) for discordance.

Eligibility Criteria
Any clinical trials or observational study designs that evaluated any digital (m/eHealth) technology with any reported outcomes (Refer to Figure 1) were included.

Data Abstraction
Two reviewers (RV, JD) independently abstracted all the data. A pre-piloted data abstraction form, was used to abstract the following items: study design, study population, sample size, digital innovation type, HIV/STIs, outcome measures (e.g. impact, acceptability and feasibility), and metrics (e.g. attendance rate, completion rate, satisfaction) (Refer to Appendix 2). We referred to a previously published framework to define and further classify the following metrics for impact, acceptability, and feasibility. 29

Subgroup Pooled Analyses
We classified study designs and then classified digital innovations into three groups: 30 a) mHealth (SMS and phone calls only; i.e. non-internet based); b) Internet-enabled mHealth and other internet-based eHealth (mobile application, website, online campaign, streamed soap opera videos, avatar-guided computer programs); c) Combined innovations (innovations that combined both mHealth (SMS/phone calls) with internet enabled m/eHealth).  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  Only one subgroup reported similar outcomes which could be pooled, SMS and phone calls, for the following outcomes: a) clinic attendance with SMS; and b) ART adherence via Medication Event Monitoring System (MEMS) caps, with SMS. We pooled these outcomes using a random effects subgroup analysis. Given the diversity in the sample populations between studies, we used the random effect meta-analysis model with DerSimonian and Laird estimator (moments method) of the between-study variance to calculate the pooled effect. We generated forest plots for visual representation of heterogeneity and pooled odds ratios (OR) with 95% confidence intervals (CI). We performed all statistical analyses using Stata/IC, version 13 (StataCorp, College Station, Texas USA). 31

Narrative Analysis
We narratively described all other data using as follows: Digital innovations were classified into the following groups based on the strength of evidence: high/strong evidence (metrics at 75-100%), moderate evidence (51-74%), and low/weak evidence (50% or less).
Acceptability: Acceptability was defined as the receptivity in using digital innovations.
Feasibility: Feasibility was defined as the perceived convenience in using digital innovations. It was reported with various metrics: completion, retention, response and referral rates.
Impact: Impact was defined as a statistically significant improvement in measured outcomes compared to a comparator group (i.e. control group or baseline observations). The metrics used to evaluated impact were: A) attendance rate, B) ART adherence, C) risk reduction, D) self-care and E) partner notification. Impact measures were evaluated on two criteria: effect size and precision. Effect size was assessed when data on a comparator group was made available. Precision of the effect estimate was assessed whenever reported, as it reflects the variance or spread of results.

Quality Assessment
We assessed study quality for both clinical trials and observational studies. We used the Cochrane Risk of Bias Tool for trials, and Newcastle-Ottawa quality assessment scale for observational studies.

Internet-based m/eHealth:
Studies evaluating internet-based eHealth innovations (21/99) reported results that were largely in favor of the following innovations: social media-based interventions for clinic attendance; avatar-guided and mobile applications for ART adherence; social media, avatar, and soap opera videos for risk reduction behaviors; mobile app for self-care.

Combined innovations:
Studies evaluating combined innovations (9/99) showed success of social media + SMS in increasing clinic attendance and partner notification; interactive websites + SMS in improving ART adherence; and mobile app + SMS in increasing self-care. Among the five impact studies, 80% reported a favorable outcome. An online campaign with SMS services increased CT, GC, and HIV tests uptake by 41%, 91%, and 190%, respectively; 123 an interactive website with SMS reminders improved ART adherence in drug-users (n=20; p=0·02); 121 a social media-based partner notification with SMS increased notified contacts by 144% (63.5% in 2011 from baseline 26% in 2010); 126 and a mobile app with SMS significantly improved self-care performance in HIV-infected individuals compared to baseline (n=19; p=0.002). 129

Quality
Studies were individually evaluated on quality criteria and biases were noted. Across trials, losses to follow-up were reported in 31% of RCTs and 55% of quasi-trials. Additionally, biases (i.e. misclassification, recall bias) were of concern in 58% of the RCTs and 64% of quasi randomized trials (Refer to Appendix 4 & 5).
In observational studies, confounding (68%) and selection bias (66%) were observed. (Refer to Appendix 6). Studies with small sample sizes, low power or insufficient follow-up time (e.g. 3 weeks or less) sometimes provided contradictory results when objective and subjective metrics evaluated the same outcome.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60   F  o  r  p  e  e  r  r  e  v  i  e  w  o  n  l  y Overall, digital innovations reported positive effects on key metrics. We noted a strong positive effect of digital innovations on clinic attendance rates (70%; 26/37), ART adherence (69%; 20/29), risk reduction behaviors (67%; 8/12) and self-care (100%; 2/2). SMS/phone calls were not able to reduce risky sexual behaviours; however social-media based interventions, particularly interactive social media, were effective in reducing risky sexual behaviors. Acceptability was found to be high for all innovations. Feasibility estimates also remained high for all innovations, except for social media-based interventions, possibly due to a perceived lack of privacy and confidentiality. Combined innovations may thus offer promise in plugging this feasibility gap, with internet-based innovations compensating for limitations in SMS-only strategies and vice versa.

Summary of findings
While mHealth (SMS/phone calls only) innovations were highly effective in improving clinic attendance, ART adherence, and turnaround time from testing to treatment, they did not report on other outcomes. It should be noted that SMS and phone calls alone failed to reduce risky sexual behaviors, which was not surprising as it is challenging to reduce risky behaviors with a prescriptive SMS alone. Population engagement is essential for risk reduction through qualitative research.
While internet-based m/eHealth innovations (social media, avatar-guided computer programs, mobile apps, and soap opera videos) demonstrated positive evidence on impact metrics, not all studies reached statistical significance. Those that failed to report a statistically significant improvement in ART adherence had small sample sizes and were underpowered to detect these outcomes (n=76 vs. n=240), and had less frequent sessions over a shorter evaluation period (2 sessions over 6 months vs. 4 sessions over 9 months). 102 107 For mobile applications, studies which reported significant effects recruited participants with varying level of adherence, 104 110 compared with studies which had high adherence at baseline (≥ 95%) and did not show significance (due to smaller changes in effect). For social media-based campaigns, the two studies that did not reach statistical significance in reducing risky sexual behaviors lacked an interactive component and simply displayed educational material, while the study that showed significant effect engaged the participants by allowing them to contact professional cognitive behavioral therapists via live chat sessions. 103 105 117 In terms of quality, confounding and selection bias were noted in observational and quasiexperimental studies, and loss to follow-up in some trials. Nevertheless, the overall validity of the findings from this review was not threatened by biases, as a large proportion of our data were derived from trials. While clinical trials were generally high quality, observational studies were medium to low quality.
Consistent reporting of metrics was lacking, which prevented a comprehensive meta-analysis. Objectives, end points, metrics, and measures, are equally important in feasibility studies and must be well designed to generate high quality evidence.
Our review is an exhaustive assessment of the role of digital innovations in improving prevention and care for HIV/STIs. Our findings resonate with many smaller systematic reviews, which have separately evaluated individual components of digital innovation, such as SMS-based mHealth. 22-23 130-137 Other systematic reviews evaluating social media-based interventions reported similar findings to ours, in improved testing uptake or improvements in sexual health. 25-27 138-139  Our review makes a valuable addition to the growing body of evidence by highlighting the success of other interactive and engaging innovations such as avatar-guided computer programs, mobile apps, streamed soap opera videos, and combined innovations. These integrated innovations and programs are gaining in popularity, because of their power to engage rural and urban audiences at many levels.
Designing combined innovations that complementarity of various media, methods, platforms, and messaging may delivery best results. This complementarity may also encourage participant engagement, to improve prevention and care metrics and measures sustainably over time. Engagement is challenging when only one innovation (e.g. mHealth SMS/phone calls only) is the sole focus, where boredom is likely.

Caveats and implications for future research
There are some caveats to considering design and evaluation of innovations. Future research needs to be focussed on tailoring innovations to the context and population, and program objectives. Innovations aiming to reduce risky sexual behaviors could be interactive and tailored to the setting and population, with a deep understanding of patients' needs and preferences. 137 140-141 Any communication with patients could be customized for timing to avoid fatigue with its uptake. For example, patients may be more responsive to weekly versus daily SMS ART reminders. 32 142 Study quality is essential to generating meaningful results. Large and representative samples of the underlying population and sound statistical techniques during data analysis or sampling methodology, can minimize selection bias. Exploring reasons for differential losses to follow-up could inform future studies. Wherever possible, a control group should be included to differentiate Hawthorne effect from the effect of the intervention. 143 Trials and impact designs can prevent or reduce confounding. Following checklists, like the one by the WHO mHealth Technical Evidence Review Group on mHealth innovations, is suggested and encouraged. 144 Objective measures (e.g. HIV/STIs diagnosis, VL load) are desired in reporting of quantitative outcomes, over subjective self-reported data (e.g. condom use, self-reported adherence). This could potentially reduce some biases (misclassification biases/ or, desirability/recall biases) that are observed with subjective reporting.
Qualitative data are rich and complement the understanding of all the contextual and population needs, and capture the dynamics of sustainability and change. They need to be integrated with quantitative data to provide a holistic picture of uptake of any digital innovation.
Quality of digital data will merit from an improvement. Across studies, a lack of integrated online impact metrics in evaluating the success of innovations was evident. With continuously evolving digital media, inventing new ways to evaluate acceptability and feasibility becomes necessary. For example, some studies tracked online metrics via Google analytics. 74 100-101 121-124 Synergy with industry powered metrics could be a new wave to measure success of digital innovations.
To scale up proven innovations, a multi-stakeholder engagement is necessary. For that, data and metrics that appeal to all sections of stakeholders will be needed. In addition to improving the

Implications for policy and practice
In consonance with other systematic reviews, evidence at-scale and over time was scarce. 138 This limits the projection of the long-term sustainability and cost effectiveness of digital innovations. More evidence on scale-up, cost savings and cost-effectiveness from Sub Saharan Africa and Asia is needed. Future investments that incentivize both: the development and evaluation of combined innovations by government and industry alike, and focus on sustainability of digital innovations with public and private partnerships, are urgently needed.

CONCLUSION
To control HIV/STIs globally, we need novel and disruptive innovations that will uniquely impact health outcomes across the spectrum of access, engagement, treatment and retention so as to impact health service delivery. On one hand, mHealth (SMS/phone calls only) and internetbased m/eHealth were found acceptable, feasible and offered complementarity in improving prevention and care of HIV/STIs. On the other hand, when combined, they provided customized and contextualized solutions for hard-to-reach populations.
Innovations need to be proven for impact and cost effectiveness, using a combination of clinical trials, quasi-randomized studies, observational studies, qualitative research studies. Integrating these innovations across various levels of healthcare with clear evaluation, monitoring, and documentation of metrics will facilitate their integration within existing health service delivery models so as to efficiently impact health outcomes over time.
Findings from this comprehensive review will be informative to all stakeholders -innovators, researchers, healthcare practitioners, policy makers and funders -worldwide seeking evidence on integrating and funding innovations, to make the entire spectrum of HIV/STI care.

ACKNOWLEDGEMENT
The authors would like to acknowledge Ms. Megan Smallwood for her assistance in editing the manuscript.
NPP: data critiquing, write-up, critique, and overall responsibility of the data.
RV, BL and SD: data synthesis, write-up and critique.
JK, TP and KD: write-up and critique. There are no conflicts of interest Data Sharing: No additional data are available. This is a systematic review/syntheses of existing studies, therefore all data are reported in the tables.

4
Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. 4 Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.

4
Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis).

4
Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators.

4
Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made.

Risk of bias in individual studies
12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was done at the study or outcome level), and how this information is to be used in any data synthesis. Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies).

5
Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified.

Study selection
17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram.

5
Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations.

Abstraction Table
Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome level assessment (see item 12).

8
Results of individual studies 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each intervention group (b) effect estimates and confidence intervals, ideally with a forest plot.

DISCUSSION
Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., healthcare providers, users, and policy makers).