Design and development

To improve engagement in HIV care and ART adherence is an mHealth intervention designed and developed in collaboration with YLWH to improve ART adherence and engagement in HIV care among this age group. WYZ design began with individual qualitative interviews with healthcare providers26 and YLWH and focus group discussions with YLWH.27 This formative research, in conjunction with the information, motivation, behavioural skills (IMB) model for understanding complex health behaviours, guided the initial WYZ design. The IMB model is based on the premise that relevant information, motivation and behavioural skills interact in dynamic ways that influence an outcome, such as improved retention in care.28 The IMB model was selected to serve as a guiding framework because it has been applied extensively in research regarding ART adherence,29–32 used successfully with youth33 34 and the components were supported by our formative research with YLWH. This conceptual design was translated into visual schematics (ie, wireframes) which were used to build the prototype (ie, WYZ V.1.0). We then used data collected from field testing this prototype to develop a beta version (WYZ V.2.0) for pilot testing.

Patient and public involvement

The iterative method used for the development of WYZ is commonly known as human-centred design (HCD).35 36 HCD focuses on creating approaches and delivering solutions to problems based on a concerted effort to understand the specific needs and perspectives of the users. In other words, one seeks to gain insight into the needs of the potential beneficiaries of an innovation and creates approaches and delivers solutions to meet their needs. HCD was selected because it allows for iterative development and refinement of a customised and tailored solution, and may increase the likelihood of user engagement with the tool. WYZ design, user testing and ongoing refinement have involved a multistep process of systematic review of the literature37; one-on-one qualitative interviews, focus groups27 and quantitative surveys with YLWH; and qualitative interviews with healthcare providers serving YLWH.26 These formative studies involved a multidisciplinary team consisting of a panel of YLWH, healthcare technology consultants, mobile developers and the research co-investigators.

We additionally used an agile methodology shown to be useful in development of software.38 In the agile methodology, the project uses incremental, iterative cycles of development commonly known as sprints and adjusts to changing requirements dynamically. At the end of each sprint, work completed during the sprint is demonstrated and submitted by the developer to the team for approval. This approach was selected because it complemented our iterative design approach, allowed the team to make immediate alterations to the application and created multiple opportunities to provide feedback throughout the course of development. The combination of HCD and an Agile software development is commonly used in industry and has recently gained traction in government and healthcare environments.39

We will continue to work with our study population to recruit and conduct the pilot study (see the Pilot study protocol section) and disseminate findings through the University of California San Francisco (UCSF) Center for AIDS Prevention Studies’ (CAPS) Community Engagement Core and the Youth Advisory Panel using presentations, community forums, journal publications and/or social media.

Architecture

We chose to build two native mobile applications (ie, iOS and Android) as opposed to one hybrid application due to improved performance and long-term viability. Native applications allow one to take advantage of existing tools available on mobile phones (eg, geolocation, camera and fingerprint log-in) without major development costs. These features allow the application to remain current and relevant in spite of changes in technologies and devices.

WYZ is composed of three primary modules, and each module contains multiple features that work together to target specific needs of the user in different situations. Table 1 provides further details of the features of WYZ.

Security and privacy

We prioritised user privacy and security in the creation of WYZ. We are leveraging a customised Salesforce Dot Com instance to serve as the backend service and storing sensitive data in a secure cloud-based database. To download the application, the user is required to satisfy a two-step authentication process: provision of a password and entry of a security code sent to the user’s registered email. WYZ is password protected and the password is required with each log-in. In case of theft, loss or misuse, study staff can remotely revoke application access. All messages or communications within the app expire after 30 days. All users use an alias and emoji instead of their name when interacting with other users within the application. An independent review conducted by the Information Technology group at the UCSF certified that the application has implemented appropriate protocols and strategies to protect users’ privacy and health information, and mitigate known security risks.

Downloading and provisioning WYZ for new users

The Hockey application, a platform that facilitates the release of application updates, is used to distribute non-publicly available applications to users, who are sent an email from the study staff containing a link for application download. This method of distribution ensures that WYZ is downloaded only on the intended devices. At first log-in, the user is prompted to set a custom passcode which is required to gain access to WYZ. In case a user forgets their passcode, the Salesforce service can be used to securely authenticate the user’s identity via their email address. To verify their identity, users are asked to enter a temporary access code sent to the user’s email.

Field testing and iterative refinement

Field testing is an important component of the iterative HCD approach. We field tested WYZ (V.1.0) with two study team members and eight YLWH who were members of the study advisory panel from October through December 2018. We asked users to log into the application at least once per week and report any bugs and application crashes, as well as suggest modifications to increase engagement. During this time frame, we checked in with users on a weekly basis and, in December 2018, we conducted a focus group with users to further discuss application refinements. We also conducted brief individual qualitative interviews with two primary care providers who provided healthcare for some of the panel members to examine the ideal experience that they would want in interacting with application users. The results of this field test were used to identify potential areas for refinements. Based on these data, an updated version of WYZ (ie, V.2.0) will be created for the forthcoming pilot study.

Study design

We will conduct a 6-month single-arm pilot study to examine WYZ feasibility and acceptability among 76 YLWH (18–29 years old) who live or receive care in the San Francisco Bay Area. The objective of this forthcoming phase is to refine the design and development and pilot test WYZ. In line with evolving guidelines from the National Institutes of Health and in the literature regarding instability of effect sizes from pilot studies and concomitant recommendations for pilot studies to eschew hypothesis testing, preliminary clinical impacts will not be examined.41 42 All recruitment and study procedures have been reviewed and approved by the UCSF Institutional Review Board. Here, we describe the pilot trial design of WYZ among 76 YLWH that will be conducted remotely.

Study settings

The study will be conducted in the San Francisco Bay Area. The research team includes a clinical pharmacist, physician, biostatistician, psychologist and a research coordinator from the UCSF CAPS. Technological support and design elements are provided by UCSF School of Medicine Technology Services and the development team consists of developers from Wolox, a technology solutions company. All study activities, including recruitment, screening, enrolment, study assessments, provision of incentives and exit interviews, will be conducted remotely.

Study participants

The potential participants will be recruited from local clinics, community-based organisations and prior research participants who have consented to be contacted for future research. The study sample will consist of 76 individuals aged 18–29 years, living with HIV and who live or receive medical care in the San Francisco Bay Area. Other inclusion criteria include ability to speak and understand English, willingness and ability to provide informed consent and access to an Android or iOS smartphone. Those with evidence of severe cognitive impairment or active psychosis that impedes their ability to provide informed consent will be excluded. Primary care providers, nurses, social workers and pharmacists who provide care to participating individuals will also be asked to participate in study activities.

Sample size justification

The sample size (n=76) was chosen to provide sufficient variability while remaining feasible in scope for a pilot study. We used NCSS PASS V.16 (Kaysville, Utah: NCSS Statistical Software) to compute (A) the distance from the mean to the 95% confidence limit for continuous feasibility and acceptability measures assuming an SD of 1.00, and (B) the exact Clopper-Pearson confidence limits and width from a sample proportion for binary feasibility and acceptability measures. The binary measure scenario further assumed a 70% feasibility/acceptability measure proportion. Assuming 20% anticipated attrition, the estimated distance from the mean to the limit is 0.26. For binary measures, the confidence limits are 0.57 and 0.81, yielding an actual width of 0.24.

Recruitment methods

A combination of recruitment strategies will be used to help achieve the study’s enrolment goals. These include strategies such as posting flyers at clinics and community-based organisations, participant submission of contact information through the study website (wyz.ucsf.edu) and contacting healthcare team members to refer application users. We will also work with a city-wide consortium of youth service providers and the CAPS Youth Advisory Panel to disseminate study information to the target population. Enrolled participants will be encouraged to refer eligible peers and receive US$10 incentive for the referee and the referred individual who participates. Lastly, participants from the principal investigator’s prior studies who had expressed interest and consented to be contacted about future research will be given the opportunity to participate, if still eligible.

Eligibility screening

Interested individuals will be provided a brief overview of the study goals and given the chance to ask questions. Individuals wishing to enrol in the study will be screened for eligibility. To confirm an individual’s age, we will require a photo identification showing their date of birth. To confirm an individual’s HIV serostatus, we will require a clinician’s letter of HIV diagnosis, copy of laboratory test results (for HIV antibody or HIV viral load), or the individual’s ART prescription showing their name. These documents can be sent as a text message to the encrypted and secure study phone for verification by study staff.

Consent and enrolment

To enrol in the study, individuals will review and sign an electronic consent form sent to them via Qualtrics (Provo, UT, USA; version March 2017). All participants will be emailed a copy of the Experimental Subject’s Bill of Rights as mandated by the State of California and a signed copy of their consent. If a participant does not have an email address, the study staff will assist them in signing up for one, as it will be necessary for gaining access to WYZ.

Participant retention

A number of steps will be taken to engage and retain participants throughout the study period. First, participants will be asked for multiple forms of contact information (including emergency contacts, clinical contacts and social media contacts) at the initial visit. They will receive monthly follow-up text messages to confirm that their contact information has not changed. Finally, participants will receive reminder text messages 24 hours before scheduled study activities.

Participant incentives

Participants can receive up to a total of US$215 incentive for completing all study activities. They will receive US$40 for the baseline session, which includes consent, application installation and baseline survey. They will then receive US$10, US$10, US$15, US$15, US$20, US$20 and US$25 for participating in check-ins at weeks 1, 2, 4, 8, 12, 16 and 20, respectively. For the final exit survey at week 24, participants will receive US$40, and those who are invited to participate in the exit qualitative interview at this time will receive an additional US$20. Part of this incentive is aimed at offsetting any data and text messaging fees associated with the application use. We estimate that participants will spend a total of 4–4.5 hours for study activities. Participants will be given a ClinCard, a reloadable debit card, which will be used for the remote provision of study incentives. Finally, we will offer US$50 to each healthcare provider for participating in a one-time qualitative interview.

Outcome measures

The main outcomes of the pilot trial include feasibility and acceptability as indicated by meeting or exceeding the benchmarks presented in table 2. Feasibility will be assessed via user metrics by examining the participant’s interactions with WYZ via a mobile analytics service called Flurry and backend (Salesforce) reporting tools (table 2). Acceptability will be examined during monthly check-ins to examine satisfaction with application use, the system usability scale (SUS),43 a study-developed acceptability survey and an exit qualitative interview. HIV viral load and CD4+ cell count will be collected to determine feasibility of collecting these measures at both baseline and study completion.

Given that application use engagement is a complex and multifactorial concept, we aim to move beyond simple metrics. Therefore, in addition to feasibility measures detailed in table 2, we will examine an engagement index (EI) that has been detailed in other mHealth interventions.44 The EI includes the following subindices: (1) click depth (number of pages a user views per session), (2) loyalty (measures how frequently users access the application during the study period), (3) recency (the time difference between each session the user accessed the application), (4) interaction (number of push notifications opened from those sent through the application), and (5) feedback (subjective measure of participants’ satisfaction with the application).

Data analysis plan

We have established minimum thresholds for determining the feasibility and acceptability of the intervention. Table 2 details these thresholds for the feasibility metrics. For acceptability, SUS scores range from 0 to 100 and scores >68 are considered above average. We will use a threshold of 70% or greater satisfaction to determine acceptability. Means and proportions will be computed at the study endpoint to enable descriptive evaluation of feasibility and acceptability. For EI, we will add all five subindices and examine cut-off points based on the distribution of the total samples’ EI scores and categorise participants as either poorly, moderately or highly engaged.

On completion of the study, up to 20 participants will be invited to complete a one-time, audio-recorded semistructured qualitative exit interview with study staff by phone to further assess the acceptability of WYZ. We will select participants based on their EI category (ie, poor, moderate or high engagement), and interviews will occur until data saturation or until 20 participants have been interviewed. The interviews will focus on the participant’s (1) attitudes towards exiting features; (2) attitudes and perceptions of privacy and information security; (3) features that they would modify, add or eliminate; and (4) suggested modifications to increase engagement and sustained use of WYZ.

Finally, we will conduct in-depth qualitative interviews with clinical team members whose patients participated in the pilot study. The interview will focus on perceived (1) burden or ease of study procedures; (2) attitudes and beliefs regarding impact of the intervention on users’ engagement in HIV care and ART adherence; (3) sustainability and scalability of the application to other youth in the clinic; (4) challenges of integrating an application into clinical work; and (5) suggested modifications to improve usability. All interviews will be done by phone, digitally recorded, transcribed verbatim and deidentified. Transcriptions will be analysed using thematic and content analysis frameworks. This combination is useful in identifying implicit and explicit ideas within the data while also allowing for quantification of qualitative data.45