Does electronic consent improve the logistics and uptake of HPV vaccination in adolescent girls? A mixed-methods theory informed evaluation of a pilot intervention

Objectives To evaluate the usability and acceptability of an electronic consent pilot intervention for school-based immunisations and assess its impact on consent form returns and human papilloma virus (HPV) vaccine uptake. Design Mixed-methods theory-informed study applying qualitative methods to examine the usability and acceptability of the intervention and quantitative methods to assess its impact. Setting and participants The intervention was piloted in 14 secondary schools in seven London boroughs in 2018. Intervention schools were matched with schools using paper consent based on the proportion of students with English as a second language and students receiving free school meals. Participants included nurses, data managers, school-link staff, parents and adolescents. Interventions An electronic consent portal where parents could record whether they agreed to or declined vaccination, and nurses could access data to help them manage the immunisation programme. Primary and secondary outcome measures Comparison of consent form return rates and HPV vaccine uptake between intervention and matched schools. Results HPV vaccination uptake did not differ between intervention and matched schools, but timely consent form return was significantly lower in intervention schools (73.3% vs 91.6%, p=0.008). The transition to using electronic consent was not straightforward, while schools and staff understood the potential benefits, they found it difficult to adapt to new ways of working which removed some level of control from schools. Reasons for lower consent form return in e-consent schools included difficulties encountered by some parents in accessing and using the intervention. Adolescents highlighted the potential for electronic consent to by-pass their information needs. Conclusions The pilot intervention did not improve consent form return or vaccine uptake due to challenges encountered in transitioning to new working practice. New technologies require embedding before they become incorporated in everyday practice. A re-evaluation once stakeholders are accustomed with electronic consent may be required to understand its impact.

I, the Submitting Author has the right to grant and does grant on behalf of all authors of the Work (as defined in the below author licence), an exclusive licence and/or a non-exclusive licence for contributions from authors who are: i) UK Crown employees; ii) where BMJ has agreed a CC-BY licence shall apply, and/or iii) in accordance with the terms applicable for US Federal Government officers or employees acting as part of their official duties; on a worldwide, perpetual, irrevocable, royalty-free basis to BMJ Publishing Group Ltd ("BMJ") its licensees and where the relevant Journal is co-owned by BMJ to the co-owners of the Journal, to publish the Work in this journal and any other BMJ products and to exploit all rights, as set out in our licence.
The Submitting Author accepts and understands that any supply made under these terms is made by BMJ to the Submitting Author unless you are acting as an employee on behalf of your employer or a postgraduate student of an affiliated institution which is paying any applicable article publishing charge ("APC") for Open Access articles. Where the Submitting Author wishes to make the Work available on an Open Access basis (and intends to pay the relevant APC), the terms of reuse of such Open Access shall be governed by a Creative Commons licence -details of these licences and which Creative Commons licence will apply to this Work are set out in our licence referred to above.
Other than as permitted in any relevant BMJ Author's Self Archiving Policies, I confirm this Work has not been accepted for publication elsewhere, is not being considered for publication elsewhere and does not duplicate material already published. I confirm all authors consent to publication of this Work and authorise the granting of this licence. To evaluate the usability and acceptability of an electronic consent intervention for school-based 29 immunisations and assess its impact on consent form returns and HPV vaccine uptake.

31
This was a mixed-methods theory-informed study which used quantitative methods to assess the 32 impact of the intervention and qualitative methods to examine its usability and acceptability.

33
Setting and participants

34
The intervention was implemented in 14 secondary schools in seven London boroughs. Intervention 35 schools were matched with schools using paper consent based on the proportion of students with 36 English as a second language and students receiving free school meals. Participants included nurses, 37 data managers, school link staff, parents and adolescents. 44 Results

45
HPV vaccination uptake did not differ between intervention and matched schools, but timely 46 consent form return was significantly lower intervention schools (73.3% (n=11) vs 91.6% (n=11), p=0.008). The transition to using electronic consent was not straightforward, whilst schools and staff 48 understood the potential benefits, they found it difficult to adapt to new ways of working which 49 removed some level of control from schools. Reasons for lower consent form return in e-consent

79
Students are typically given a paper consent form that their parents/legal guardians need to sign.

80
The passage of the form from teachers to students to parents and back again can impact on form 81 return and have a detrimental effect on vaccine uptake [4][5][6][7]. Follow-up by school staff and 82 immunisation teams improves uptake but is resource intensive [8,9]. In the context of the drive   We compared e-consent schools with paper consent schools in terms of timely form return, 148 outcome of consent and vaccination uptake using Kruskall Wallis tests for statistical significance.

149
Where data was missing from a school that school and its matched pair were excluded from the 150 analysis. All analyses were done in MS Excel and Stata 15.  Roman Catholic (all paper consent), three Church of England (two e-consent, one paper consent) and 224 one was another Christian faith school (paper consent).

225
The proportion of pupils eligible for free school meals, with English as an additional language and 226 students' ethnicity profile was similar between the e-consent and paper consent schools (Table 1). Return of consent forms ahead of session 235 Overall 83% of consent forms (paper or e-consent) were returned prior to the vaccination session.

236
However, compared with paper schools timely (prior to the planned session) return was lower in the 237 e-consent schools (73.3% (n=11) vs 91.6% (n=11), p=0.008). We could only measure this difference in 238 22 matched schools.

239
Outcome of consent 240 There was no statistically significant difference in the proportion of pupils for whom a "yes" consent 241 was received (prior to or on the day of the session) between the paper (n=14) and e-consent (n=14) 242 schools (85% in e-consent schools, 83% in paper consent schools, p=0.89).

243
Vaccination uptake 244 There was no statistically significant difference in the proportion of pupils that were vaccinated at 245 the scheduled vaccination session between the paper (n=14) and e-consent (n=14) schools (80.6% vs  expressed reservations about not able to review paper consent forms prior to immunisation. Due to 261 tight deadlines only one orientation session took place before the e-consent intervention was 262 introduced, which meant that the bulk of learning happened on the job.

263
"I think as well, it was probably four days before our first session, we didn't know what we were 264 doing...so I do feel we are running before we can walk." (Immunisation Team 2) 265 Intervention not fully operational 266 267 The data platform component of the online portal was not operational prior to implementation.

268
Parents were able to access and complete the e-consent form, but the immunisation teams could 269 not review student's online consent forms or upload data during immunisation sessions. Instead 270 large (A3) paper sheets with information about who had provided consent were prepared by the 271 data managers. The sheets were difficult for to decipher during busy sessions and nurses were less 272 able to prepare cohort figures and tally sheets in advance.

273
"It was an anti-climax not being able to use the laptops and still have a paper sheet in front of me."  immunisation teams provided schools with details of non-responders. In this case follow-up could be 292 more targeted and involve text messages and phone calls as wells as emails. One school used a 293 translator to engage parents who did not understand the consent process due to language barriers.

294
Another school was not willing to send out emails and asked the immunisation team to provide 295 them with printed letters referring to the weblink to send to parents.

296
Of the seven e-consent schools who completed the feedback form, four were positive about the 297 intervention and how it had been implemented stating that it had reduced their workload. Another 298 school was mainly positive but noted that some parents had found the e-consent form difficult to 299 access, another reported that their parent cohort had found the system very difficult to access and 300 use, and the last school was the one who had used letters to disseminate the weblink.

301
Pathways 302 303 Navigating the e-consent form and related information 304 305 The e-consent form included links to an HPV vaccination leaflet. However, none of the interviewees 306 had downloaded or read this leaflet for the following reasons: accessed information elsewhere, 307 already sufficiently informed, older daughter vaccinated, positive about vaccination.

308
Proactive information seeking was more common in families who were vaccine hesitant. Parents 309 who were more confident about vaccines restricted their information seeking to NHS sources and 310 suggested that a 'road map' to adolescent vaccination could be useful. Adolescents reported a variety of information seeking behaviours. Some just accepted HPV 312 vaccination as 'something that needs to be done' and felt reassured that it was recommended by the 313 NHS: "I think because it's like by the NHS -it kind of gives it validation." (Adolescent 9 -Yes). Others 314 wanted the HPV vaccine leaflet to include more information about HPV and related health risks and 315 vaccine side-effects, so that they did not panic if they experienced any of these.

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school has been involved in using the e-consent system as part of the HPV vaccination programme and we are keen to learn from this experience. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school is likely to use this system in the future hence we are keen to learn from your current experience of being involved in coordinating the HPV vaccination programme for adolescent girls. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.
The information that you share with us will help us to evaluate the usability and acceptability of the e-consent system for school-based immunisations and assess its effectiveness in promoting consent form return and HPV vaccine uptake. The findings will guide decision-making about the future use of the e-consent system and help improve the system.

Acceptability of the e-consent system
 What were your initial thoughts about the e-consent system? (Probes: concerns, hopes, expectations)  What did your colleagues think about the system?  How acceptable do you think the system is for parents, teenagers, and school staff involved in the programme?
Usability of the e-consent system  How helpful was the training in preparing you to use the e-consent system?  What was your experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Triage of consent forms o Use of the system during immunisation sessions in schools o Uploading data to Trust databases and Child Health Immunisation Systems o Any other stages  What contact did you have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?  What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?  How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?  What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
 What is your experience of working with schools in organising immunisation sessions?  What were schools reactions to the introduction of the e-consent system?  How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?) How does this compare with the paper-based system?  What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?  Have schools conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you been involved in these?

Adolescent self-consent
 What do you think about giving teen-agers the opportunity to self-consent for vaccination? was the procedure, and how did you assess for competency?  How does the e-consent system support self-consent? What is the procedure for this? Did any teen-agers use the e-consent system to give consent for HPV vaccination? What was that experience like? well, what needs to be changed?)  How would you modify and improve the e-consent system based on the piloting experience?  What is your advice for other immunisation providers thinking about implementing econsent?  Has the e-consent system achieved its intended outcomes from your perspective?

Do you have any other comments?
Thank you for your time. Decision to launch the e-consent system  What were the main reasons you decided to implement the e-consent system?  How did you decide who to collaborate with in developing the e-consent system?  How was the process of development? What went well, what was difficult?  How did your test the system prior to the piloting phase?
Acceptability of the e-consent system  How did your team initially react to the plans to introduce the e-consent system? (Probes: Expectations, concerns, questions)  Did opinion change over time, and if yes, how?  How acceptable do you think the e-consent system is for nurses, parents, teenagers, and school staff involved in the programme?
Usability of the e-consent system  How were you involved in using the e-consent system?  What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages? o Triage of consent forms o Use of the system during immunisation sessions in schools o Uploading data to Trust databases and Child Health Immunisation Systems o Any other stages  What contact did you/your team have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?  What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?  How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?  What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
 What is your experience of working with schools in organising immunisation sessions?  What were schools reactions to the introduction of the e-consent system? How did you inform them about the e-consent system, what guidance or training did you provide?  How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?)  How does this compare with schools that are using the paper system?  What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?  Have schools (those using e-consent and paper system) conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you/your team been involved in these?

Adolescent self-consent
 Please describe the process you have used for self-consent  What do you think about giving teenagers the opportunity to self-consent for vaccination? well, what needs to be changed?)  How will you modify and improve the e-consent system based on the piloting experience?  What have been the costs of implementing the e-consent system?  What is your advice for other immunisation providers thinking about implementing econsent?  Has the e-consent system achieved its intended outcomes from your perspective?

Do you have any other comments?
Thank you for your time. Decision to launch the e-consent system (for senior administrators)  What were the main reasons the Trust decided to implement the e-consent system?  How was the process of development? What went well, what was difficult?  How did your test the system prior to the piloting phase?

Acceptability of the e-consent system
 What were your initial thoughts about the plans to introduce the e-consent system?
(Probes: Expectations, concerns, questions)  Did your opinion change over time, and if yes, how?  What is your current view on the acceptability of the e-consent system? Usability of the e-consent system  How are you involved in using the e-consent system?  What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Administration of data from the consent forms o Administration of data from school and catch-up immunisation sessions o Uploading data to Trust databases and Child Health Immunisation Systems o Sharing data with GPs o Any other stages  What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
 What role do you play in collaborating with schools to organising immunisation sessions?  What were schools reactions to the introduction of the e-consent system? How were they informed about the e-consent system, what guidance or training did they receive?  How has the e-consent system supported interactions between schools and immunisation providers? (Probes: what has worked well, what has been challenging?)  How does this compare with schools that are using the paper system?  What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system? well, what needs to be changed?)  How will you modify and improve the e-consent system based on the piloting experience?  What have been the costs of implementing the e-consent system?  What is your advice for other immunisation providers thinking about implementing econsent?  Has the e-consent system achieved its intended outcomes from your perspective?

Any other comments
Do you have any other comments?
Thank you for your time. Usability of the e-consent system  How did you find using the e-consent system? Who completed the form?  Was it easy to access the e-consent site via the email sent from school?  What was you experience of completing the consent form online? Were there any parts that you found difficult to complete? How did you know if you had completed the form properly?  What was your experience of accessing the HPV vaccination programme leaflets via the econsent system? Did you read these online or did you print them out? Did you share these leaflets with your teenager or other members of the family?  Did you contact the immunisation team for help to complete the e-consent form? What support did you require? How easy was it to contact them?  How do you think that the e-consent form could be improved?

Acceptability of the e-consent system
 What are your views on giving consent electronically for your teenagers' vaccinations?
(Probes: any concerns about sharing personal data online?)  What did your teen-ager think about this method of giving consent?  What are your views on how the e-consent system supports information sharing and communication about teenage vaccination? (Probes: what went well, what did they find difficult)  How well prepared do you think your teenager was prior to the immunisation session?
(Probes: access to information, educational sessions in school)

Views on adolescent self-consent
 What do you think about the option of teen-agers providing self-consent for vaccination?  What is your understanding of self-consent in adolescence?  Would you be happy for your teen-ager to give self-consent for vaccination? If yes, why. If no, why?

Any other comments
Do you have any other thoughts about adolescent vaccination and how consent is obtained for this?
Thank you for your time.  Usability of the e-consent system  Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?  What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?  Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents)  What do your friends think about the e-consent system?  How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
 What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?)  What are your views on how the e-consent system supports finding out about and talking about teenage vaccination?  How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it)  Were you given the opportunity to ask any questions before you were vaccinated?  Had you participated in any information sessions about vaccination at school before you were vaccinated? If yes, what were these and what did you learn?  Had you received any other information about the HPV vaccine before you were vaccinated?

Experience of the immunisation session
If yes, who gave you this?

Views on adolescent self-consent
 What do you think about the option of teen-agers providing self-consent for vaccination?  What is your understanding of self-consent?  Have you ever been given the option to provide self-consent for vaccination?  Would you be happy to provide self-consent for vaccination? If yes, why. If no, why?

Any other comments
Do you have any other thoughts about vaccination and how consent is obtained for this?  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  Usability of the e-consent system  Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?  What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?  Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents)  What do your friends think about the e-consent system?  How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
 What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?)  What did your parents think about this way of obtaining consent?

Experience of the immunisation session
 Could you tell me a bit about your experience of receiving the HPV vaccine in school? What was the session like, how did you feel, what went well, what was difficult?  How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it)  Were you given the opportunity to ask any questions before you were vaccinated? To evaluate the usability and acceptability of an electronic consent pilot intervention for school-29 based immunisations and assess its impact on consent form returns and HPV vaccine uptake.

31
Mixed-methods theory-informed study applying qualitative methods to examine the usability and 32 acceptability of the intervention and quantitative methods to assess its impact.

33
Setting and participants

34
The intervention was piloted in 14 secondary schools in seven London boroughs in 2018.

35
Intervention schools were matched with schools using paper consent based on the proportion of 36 students with English as a second language and students receiving free school meals. Participants 37 included nurses, data managers, school-link staff, parents and adolescents.

39
An electronic consent portal where parents could record whether they agreed to declined 40 vaccination and nurses could access data to help them manage the immunisation programme.

42
Comparison of consent form return rates and HPV vaccine uptake between intervention and 43 matched schools.

45
HPV vaccination uptake did not differ between intervention and matched schools, but timely 46 consent form return was significantly lower in intervention schools (73.3% (n=11) vs 91.6% (n=11), p=0.008). The transition to using electronic consent was not straightforward, whilst schools and staff 48 understood the potential benefits, they found it difficult to adapt to new ways of working which 49 removed some level of control from schools. Reasons for lower consent form return in e-consent

119
The reason that the quantitative analysis only focused on Year 1 was that the schools that received 120 the intervention between Year 1 & 2 changed. At each vaccination session nurses completed a "tally sheet", with details of the consents received 133 prior to or during the session, any absences and the number of vaccinations given. These tally 134 sheets were completed for paper and e-consent schools.

135
In addition, we extracted the name of the school, date and time of consent form completion and 136 type of consent (agreement by parent or adolescent self-consent or decline) for each consent form 137 in the e-consent system. This non-identifiable information was combined with tally sheet data, 138 which was manually transcribed into MS Excel. Where there were discrepancies or missing data this 139 was checked with the immunisation teams and in the case of the e-consent schools the system data 140 was used in preference to the tally sheet data.

141
Data analysis 142 Characteristics of the paper and e-consent schools were described in terms of proportion of pupils 143 receiving free school meals, speaking English as an additional language, ethnicity and characteristics 144 of the schools (religious affiliation and state/privately funded), using data from the Office of National

147
For both paper and e-consent schools, we calculated: (i) the proportion of the pupils for whom a 148 consent form had not been returned prior to the vaccination session, (ii) the proportion of pupils 149 vaccinated at the planned session and (iii) the proportion of pupils for who a "yes" consent was 150 received (prior to or on the day of the planned vaccination session).

151
We compared e-consent schools with paper consent schools in terms of timely form return, 152 outcome of consent and vaccination uptake using Kruskall Wallis tests for statistical significance.

153
Where data was missing from a school that school and its matched pair were excluded from the 168 School feedback forms from paper and e-consent schools    The SSIs and FGD were audio recorded with participant's permission and transcribed anonymously.

206
Transcripts, observational field notes and school feedback forms were downloaded into a qualitative 207 data analysis management software programme (NVivo 12). We adopted a thematic analytical 208 approach which combined semi-deductive mapping of data to the 'input', 'activities', 'pathways', Roman Catholic (all paper consent), three Church of England (two e-consent, one paper consent) and 225 one was another Christian faith school (paper consent).

226
The proportion of pupils eligible for free school meals, with English as an additional language and 227 students' ethnicity profile was similar between the e-consent and paper consent schools (Table 1). Using the e-consent intervention 338 The parents we interviewed found the system easy to use and usually completed the form as soon as

Heads up as what to expect
 Ensured daughter knew what to expect; this exchange usually occurred shortly before the immunisation session;  Offered to answer any questions depending on adolescents' desire to know more.

More in-depth discussions
 Joint decision-making between parents and adolescents about the important of vaccination, this sometimes involved accessing additional information.   to find the right level of involvement  Ensuring students receive a paper copy of the HPV adolescent programme leaflet produced by Public Health England in addition to information provided in the e-consent form  Pushing more for assemblies and contact with adolescent girls prior to the immunisation sessions  Providing ongoing training and mentoring of immunisation teams on use of the intervention

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school has been involved in using the e-consent system as part of the HPV vaccination programme and we are keen to learn from this experience. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.
The information that you share with us will help us to evaluate the usability and acceptability of the e-consent system for school-based immunisations and assess its effectiveness in promoting consent form return and HPV vaccine uptake. The findings will guide decision-making about the future use of the e-consent system and help improve the system.

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school is likely to use this system in the future hence we are keen to learn from your current experience of being involved in coordinating the HPV vaccination programme for adolescent girls. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.
The information that you share with us will help us to evaluate the usability and acceptability of the e-consent system for school-based immunisations and assess its effectiveness in promoting consent form return and HPV vaccine uptake. The findings will guide decision-making about the future use of the e-consent system and help improve the system.

Experience of obtaining consent in school-based vaccination programmes
• Could you tell me about your experience of obtaining consent from parents and adolescents for school-aged vaccinations?
o What has worked well and what has been challenging? o How have you overcome challenges in the past?

Acceptability of the e-consent system
• What were your initial thoughts about the e-consent system? (Probes: concerns, hopes, expectations) • What did your colleagues think about the system? • How acceptable do you think the system is for parents, teenagers, and school staff involved in the programme?

Usability of the e-consent system
• How helpful was the training in preparing you to use the e-consent system?
• What was your experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Triage of consent forms o Use of the system during immunisation sessions in schools o Uploading data to Trust databases and Child Health Immunisation Systems o Any other stages • What contact did you have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?
• What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?
• How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?
• What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
• What is your experience of working with schools in organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? • How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?) How does this compare with the paper-based system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?
• Have schools conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you been involved in these?

Adolescent self-consent
• What do you think about giving teen-agers the opportunity to self-consent for vaccination? (Probes: age, what vaccines e.g. HPV) • Have you obtained self-consent for adolescent immunisation? If yes, in what instances, what was the procedure, and how did you assess for competency?
• How does the e-consent system support self-consent? What is the procedure for this? Did any teen-agers use the e-consent system to give consent for HPV vaccination? What was that experience like?
• Have you had conversations with schools about adolescent self-consent? What are their views?
• Have you had conversations with parents/guardians about adolescent self-consent? What were their views?

Reflections of the use of the e-consent system
• What have been the main lessons in piloting the e-consent system (Probes: what has worked well, what needs to be changed?) • How would you modify and improve the e-consent system based on the piloting experience?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Do you have any other comments?
Thank you for your time.

Decision to launch the e-consent system
• What were the main reasons you decided to implement the e-consent system?
• How did you decide who to collaborate with in developing the e-consent system?
• How was the process of development? What went well, what was difficult?
• How did your test the system prior to the piloting phase?

Acceptability of the e-consent system
• How did your team initially react to the plans to introduce the e-consent system? (Probes: Expectations, concerns, questions) • Did opinion change over time, and if yes, how?
• How acceptable do you think the e-consent system is for nurses, parents, teenagers, and school staff involved in the programme?

Usability of the e-consent system
• How were you involved in using the e-consent system?
• What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Triage of consent forms o Use of the system during immunisation sessions in schools o Uploading data to Trust databases and Child Health Immunisation Systems o Any other stages • What contact did you/your team have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?
• What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?
• How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?
• What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
• What is your experience of working with schools in organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? How did you inform them about the e-consent system, what guidance or training did you provide?
• How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?) • How does this compare with schools that are using the paper system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?
• Have schools (those using e-consent and paper system) conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you/your team been involved in these?

Adolescent self-consent
• Please describe the process you have used for self-consent • What do you think about giving teenagers the opportunity to self-consent for vaccination? (Probes: age, what vaccines e.g. HPV) • Have you had conversations with schools about adolescent self-consent? What are their views?
• Have you had conversations with parents/guardians about adolescent self-consent? What were their views?

Reflections of the use of the e-consent system
• What have been the main lessons in piloting the e-consent system (Probes: what has worked well, what needs to be changed?) • How will you modify and improve the e-consent system based on the piloting experience?
• What have been the costs of implementing the e-consent system?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Do you have any other comments?
Thank you for your time.

Decision to launch the e-consent system (for senior administrators)
• What were the main reasons the Trust decided to implement the e-consent system?
• How was the process of development? What went well, what was difficult?
• How did your test the system prior to the piloting phase?

Acceptability of the e-consent system
• What were your initial thoughts about the plans to introduce the e-consent system? (Probes: Expectations, concerns, questions) • Did your opinion change over time, and if yes, how?
• What is your current view on the acceptability of the e-consent system?

Usability of the e-consent system
• How are you involved in using the e-consent system?
• What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Administration of data from the consent forms o Administration of data from school and catch-up immunisation sessions o Uploading data to Trust databases and Child Health Immunisation Systems o Sharing data with GPs o Any other stages • What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
• What role do you play in collaborating with schools to organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? How were they informed about the e-consent system, what guidance or training did they receive?
• How has the e-consent system supported interactions between schools and immunisation providers? (Probes: what has worked well, what has been challenging?) • How does this compare with schools that are using the paper system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system? • How will you modify and improve the e-consent system based on the piloting experience?
• What have been the costs of implementing the e-consent system?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Any other comments
Do you have any other comments?
Thank you for your time. • How did you involve your teenager in this decision-making? Were you in agreement about having the vaccine?
• Have you ever changed your mind about providing consent for a vaccination, if yes why and what did you do?

Usability of the e-consent system
• How did you find using the e-consent system? Who completed the form?
• Was it easy to access the e-consent site via the email sent from school?
• What was you experience of completing the consent form online? Were there any parts that you found difficult to complete? How did you know if you had completed the form properly?
• What was your experience of accessing the HPV vaccination programme leaflets via the econsent system? Did you read these online or did you print them out? Did you share these leaflets with your teenager or other members of the family?
• Did you contact the immunisation team for help to complete the e-consent form? What support did you require? How easy was it to contact them?
• How do you think that the e-consent form could be improved?

Acceptability of the e-consent system
• What are your views on giving consent electronically for your teenagers' vaccinations? (Probes: any concerns about sharing personal data online?) • What did your teen-ager think about this method of giving consent?
• What are your views on how the e-consent system supports information sharing and communication about teenage vaccination? • How well prepared do you think your teenager was prior to the immunisation session? (Probes: access to information, educational sessions in school)

Views on adolescent self-consent
• What do you think about the option of teen-agers providing self-consent for vaccination?
• What is your understanding of self-consent in adolescence?
• Would you be happy for your teen-ager to give self-consent for vaccination? If yes, why. If no, why?

Do you have any other thoughts about adolescent vaccination and how consent is obtained for this?
Thank you for your time.

Experience of giving consent for vaccination
• Who decided whether you would receive the HPV vaccine? How did they make this decision?
• How were you involved in this decision-making? Were you in agreement with your parents about this decision?

Usability of the e-consent system
• Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?
• What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?
• Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents) • What do your friends think about the e-consent system? • How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
• What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?) • What are your views on how the e-consent system supports finding out about and talking about teenage vaccination? • How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it)

Experience of the immunisation session
• Were you given the opportunity to ask any questions before you were vaccinated?
• Had you participated in any information sessions about vaccination at school before you were vaccinated? If yes, what were these and what did you learn?
• Had you received any other information about the HPV vaccine before you were vaccinated? If yes, who gave you this?

Views on adolescent self-consent
• What do you think about the option of teen-agers providing self-consent for vaccination?
• What is your understanding of self-consent?
• Have you ever been given the option to provide self-consent for vaccination?
• Would you be happy to provide self-consent for vaccination? If yes, why. If no, why?

Any other comments
Do you have any other thoughts about vaccination and how consent is obtained for this?

Experience of giving consent for vaccination
• Who decided whether you would receive the HPV vaccine?
• How were you involved in this decision-making? Were you in agreement with your parents about this decision?

Usability of the e-consent system
• Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?
• What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?
• Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents) • What do your friends think about the e-consent system?
• How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
• What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?) • What did your parents think about this way of obtaining consent?

Experience of the immunisation session
• Could you tell me a bit about your experience of receiving the HPV vaccine in school? What was the session like, how did you feel, what went well, what was difficult?
• How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it) • Were you given the opportunity to ask any questions before you were vaccinated? Accessed via: https://www.equator-network.org/reporting-guidelines/the-quality-of-mixedmethods-studies-in-health-services-research/ Applied to:

DOES ELECTRONIC CONSENT IMPROVE THE LOGISTICS AND UPTAKE OF HPV VACCINATION IN ADOLESCENT GIRLS? A MIXED METHODS THEORY INFORMED EVALUATION OF A PILOT INTERVENTION
(1) Describe the justification for using a mixed methods approach to the research question.
The use of a mixed methods theory-informed study design is described and justified in the strengths and limitations summary box (page 3, lines 60-62) and the study design section (page 5, lines 112-120). It is also illustrated in Figure 2. The study design allowed us to measure the effect of a pilot e-consent intervention on immunisation performance and identify mechanisms that facilitated or impeded implementation.
(2) Describe the design in terms of the purpose, priority and sequence of methods.
This is described in the study design section (page 5, lines 112-120); the choice of methods was informed by the need to obtain qualitative data to examine the implementation process and quantitative data to assess outcomes and the impact of the intervention.
(3) Describe each method in terms of sampling, data collection and analysis.
These are described in full in the methods section of the manuscript. The sampling frame for the schools participating in the evaluation is detailed in the selection of schools. The boundaries sampling of interviews participants for the qualitative methods were set by this selection. We interviewed all the staff who were actively involved in the implementation process and observed immunisation sessions at schools in each of the boroughs that piloted the intervention. The recruitment and sampling of parent and adolescent interviewees (individual and paired parent/adolescent) is outlined in lines 182-188. We were only able to recruit parents and by proxy adolescents who had ticked a box on the e-consent portal stating they were willing to be interviewed. Informed consent was obtained before all qualitative interviews as described in the ethics section at the end of the manuscript.
(4) Describe where integration has occurred, how it has occurred and who has participated in it. The study team (co-authors) met every 3-4 months to discuss the results of the evaluation and compare quantitative with qualitative findings. The qualitative findings were pivotal for understanding why the pilot intervention did not achieve the expected improvement to consent form return in Year 1 and to identify mechanisms that facilitated or impeded the implementation of the intervention. Collaboration between implementers and evaluators in this real time evaluation also supported the iterative development of the pilot intervention, which is critical to the longer-term integration of new working practices. (6) Describe any insights gained from mixing or integrating methods.
As stated in point one the use of diverse methods allowed us to examine the implementation of the pilot intervention and assess its effect on consent form returns (outcomes) and HPV vaccine uptake (impact). Using qualitative methods over a period of a year was very useful to trace how the intervention was adapted over time to improve its acceptability and usability. The design of our evaluation could have been improved if we had been able to repeat the quantitative data collection in Year 2. This was not possible since the new iteration of the intervention was not ready when the original sample of e-consent schools were scheduled to receive the HPV vaccine in Year2. This is a constraint of evaluating interventions in real-time. To evaluate the usability and acceptability of an electronic consent pilot intervention for school-29 based immunisations and assess its impact on consent form returns and HPV vaccine uptake.

31
Mixed-methods theory-informed study applying qualitative methods to examine the usability and 32 acceptability of the intervention and quantitative methods to assess its impact.

33
Setting and participants

34
The intervention was piloted in 14 secondary schools in seven London boroughs in 2018.

35
Intervention schools were matched with schools using paper consent based on the proportion of 36 students with English as a second language and students receiving free school meals. Participants 37 included nurses, data managers, school-link staff, parents and adolescents.

39
An electronic consent portal where parents could record whether they agreed or declined 40 vaccination and nurses could access data to help them manage the immunisation programme.

42
Comparison of consent form return rates and HPV vaccine uptake between intervention and 43 matched schools.

151
We compared e-consent schools with paper consent schools in terms of timely form return, 152 outcome of consent and vaccination uptake using Kruskall Wallis tests for statistical significance.

235
However, amongst the 22 matched schools where this data was available, compared with paper 236 schools timely (prior to the planned session) return was lower in the e-consent schools (73.3% vs 237 91.6%, p=0.008).

267
Parents were able to access and complete the e-consent form, but the immunisation teams could

284
Paper-consent schools could monitor this directly by counting forms, but with e-consent schools

Heads up as what to expect
 Ensured daughter knew what to expect; this exchange usually occurred shortly before the immunisation session;  Offered to answer any questions depending on adolescents' desire to know more.

More in-depth discussions
 Joint decision-making between parents and adolescents about the important of vaccination, this sometimes involved accessing additional information.   to find the right level of involvement  Ensuring students receive a paper copy of the HPV adolescent programme leaflet produced by Public Health England in addition to information provided in the e-consent form  Pushing more for assemblies and contact with adolescent girls prior to the immunisation sessions  Providing ongoing training and mentoring of immunisation teams on use of the intervention    1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60 For peer review only -http://bmjopen.bmj.com/site/about/guidelines.xhtml

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school has been involved in using the e-consent system as part of the HPV vaccination programme and we are keen to learn from this experience. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.

Introduction
The Hounslow and Richmond Community Health Trust is in the process of introducing a system that facilitates the collection of electronic consent for school-aged vaccinations. Your school is likely to use this system in the future hence we are keen to learn from your current experience of being involved in coordinating the HPV vaccination programme for adolescent girls. We would be very grateful if you could complete this anonymous evaluation form and return it in the attached envelope to a member of the immunisation team at the end of the immunisation session today.

Acceptability of the e-consent system
• What were your initial thoughts about the e-consent system? (Probes: concerns, hopes, expectations) • What did your colleagues think about the system? • How acceptable do you think the system is for parents, teenagers, and school staff involved in the programme?

Usability of the e-consent system
• How helpful was the training in preparing you to use the e-consent system?
• What was your experience of using the e • What contact did you have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?
• What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?
• How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?

Interaction between schools and immunisation teams
• What is your experience of working with schools in organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? • How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?) How does this compare with the paper-based system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?
• Have schools conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you been involved in these?

Adolescent self-consent
• What do you think about giving teen-agers the opportunity to self-consent for vaccination? (Probes: age, what vaccines e.g. HPV) • Have you obtained self-consent for adolescent immunisation? If yes, in what instances, what was the procedure, and how did you assess for competency?
• How does the e-consent system support self-consent? What is the procedure for this? Did any teen-agers use the e-consent system to give consent for HPV vaccination? What was that experience like?
• Have you had conversations with schools about adolescent self-consent? What are their views?
• Have you had conversations with parents/guardians about adolescent self-consent? What were their views?

Reflections of the use of the e-consent system
• What have been the main lessons in piloting the e-consent system (Probes: what has worked well, what needs to be changed?) • How would you modify and improve the e-consent system based on the piloting experience?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Any other comments
Do you have any other comments?
• How did you decide who to collaborate with in developing the e-consent system?
• How was the process of development? What went well, what was difficult?
• How did your test the system prior to the piloting phase?

Acceptability of the e-consent system
• How did your team initially react to the plans to introduce the e-consent system? (Probes: Expectations, concerns, questions) • Did opinion change over time, and if yes, how?
• How acceptable do you think the e-consent system is for nurses, parents, teenagers, and school staff involved in the programme?

Usability of the e-consent system
• How were you involved in using the e-consent system?
• What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Triage of consent forms o Use of the system during immunisation sessions in schools o Uploading data to Trust databases and Child Health Immunisation Systems o Any other stages • What contact did you/your team have with parents about the use of the e-consent system? How did parents find using the e-consent system? What did they find difficult?
• What do you think about how the system may support communication between parents/guardians and adolescents about immunisation? How does this compare with the paper-based system?
• How does the e-consent system support communication between immunisation teams, parents and students? How does this compare with the paper-based system?
• What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
• What is your experience of working with schools in organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? How did you inform them about the e-consent system, what guidance or training did you provide?
• How has the e-consent system supported interactions between schools and immunisation teams? (Probes: what has worked well, what has been challenging?) • How does this compare with schools that are using the paper system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system?
• Have schools (those using e-consent and paper system) conducted any educational activities in preparation for the HPV immunisation visits? What were these? How have you/your team been involved in these?

Adolescent self-consent
• Please describe the process you have used for self-consent • What do you think about giving teenagers the opportunity to self-consent for vaccination? (Probes: age, what vaccines e.g. HPV) • Have you had conversations with schools about adolescent self-consent? What are their views?
• Have you had conversations with parents/guardians about adolescent self-consent? What were their views?

Reflections of the use of the e-consent system
• What have been the main lessons in piloting the e-consent system (Probes: what has worked well, what needs to be changed?) • How will you modify and improve the e-consent system based on the piloting experience?
• What have been the costs of implementing the e-consent system?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Any other comments
Do you have any other comments?
Thank you for your time. Decision to launch the e-consent system (for senior administrators) • What were the main reasons the Trust decided to implement the e-consent system?
• How was the process of development? What went well, what was difficult?
• How did your test the system prior to the piloting phase?

Acceptability of the e-consent system
• What were your initial thoughts about the plans to introduce the e-consent system? (Probes: Expectations, concerns, questions) • Did your opinion change over time, and if yes, how?
• What is your current view on the acceptability of the e-consent system?

Usability of the e-consent system
• How are you involved in using the e-consent system?
• What was your/your teams' experience of using the e-consent system? What worked well, what was difficult in the following stages?
o Administration of data from the consent forms o Administration of data from school and catch-up immunisation sessions o Uploading data to Trust databases and Child Health Immunisation Systems o Sharing data with GPs o Any other stages • What are the advantages and disadvantaged of the e-consent system in comparison with the paper-based system?

Interaction between schools and immunisation teams
• What role do you play in collaborating with schools to organising immunisation sessions?
• What were schools reactions to the introduction of the e-consent system? How were they informed about the e-consent system, what guidance or training did they receive?
• How has the e-consent system supported interactions between schools and immunisation providers? (Probes: what has worked well, what has been challenging?) • How does this compare with schools that are using the paper system?
• What have schools told you about their experience of using the e-consent system? How do they compare it with the paper-based system? • How will you modify and improve the e-consent system based on the piloting experience?
• What have been the costs of implementing the e-consent system?
• What is your advice for other immunisation providers thinking about implementing econsent?
• Has the e-consent system achieved its intended outcomes from your perspective?

Any other comments
Do you have any other comments?
Thank you for your time. • What is your view about adolescent vaccination? Are there any vaccines that you are concerned about?

Experience of providing consent for adolescent vaccinations
• Could you tell me about your experience of providing consent for your teenager(s) to be vaccinated?
• How did you make your decision about providing consent? o Who else did you talk to about this? What did you discuss? What were their views? (Probes: Partner, teenager, other family members, friends, health professionals) • How did you involve your teenager in this decision-making? Were you in agreement about having the vaccine?
• Have you ever changed your mind about providing consent for a vaccination, if yes why and what did you do?

Usability of the e-consent system
• How did you find using the e-consent system? Who completed the form?
• Was it easy to access the e-consent site via the email sent from school?
• What was you experience of completing the consent form online? Were there any parts that you found difficult to complete? How did you know if you had completed the form properly?
• What was your experience of accessing the HPV vaccination programme leaflets via the econsent system? Did you read these online or did you print them out? Did you share these leaflets with your teenager or other members of the family?
• Did you contact the immunisation team for help to complete the e-consent form? What support did you require? How easy was it to contact them?
• How do you think that the e-consent form could be improved?

Acceptability of the e-consent system
• What are your views on giving consent electronically for your teenagers' vaccinations? (Probes: any concerns about sharing personal data online?) • What did your teen-ager think about this method of giving consent?
• What are your views on how the e-consent system supports information sharing and communication about teenage vaccination? • How well prepared do you think your teenager was prior to the immunisation session? (Probes: access to information, educational sessions in school)

Views on adolescent self-consent
• What do you think about the option of teen-agers providing self-consent for vaccination?
• What is your understanding of self-consent in adolescence?
• Would you be happy for your teen-ager to give self-consent for vaccination? If yes, why. If no, why?

Any other comments
Do you have any other thoughts about adolescent vaccination and how consent is obtained for this?
Thank you for your time. • What is your view about the HPV vaccination? (Probes: what is good about it, any concerns about the vaccine or the process of vaccination) • Who have you talked to about having the HPV vaccine? (Probes: parents, friends, school nurse, other health professionals, other family members) What did you discuss?

Experience of giving consent for vaccination
• Who decided whether you would receive the HPV vaccine? How did they make this decision?
• How were you involved in this decision-making? Were you in agreement with your parents about this decision?

Usability of the e-consent system
• Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?
• What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?
• Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents) • What do your friends think about the e-consent system? • How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
• What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?) • What are your views on how the e-consent system supports finding out about and talking about teenage vaccination? • How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it)

Experience of the immunisation session
• Were you given the opportunity to ask any questions before you were vaccinated?
• Had you participated in any information sessions about vaccination at school before you were vaccinated? If yes, what were these and what did you learn?
• Had you received any other information about the HPV vaccine before you were vaccinated? If yes, who gave you this?

Views on adolescent self-consent
• What do you think about the option of teen-agers providing self-consent for vaccination?
• What is your understanding of self-consent?
• Have you ever been given the option to provide self-consent for vaccination?
• Would you be happy to provide self-consent for vaccination? If yes, why. If no, why?

Any other comments
Do you have any other thoughts about vaccination and how consent is obtained for this? • What is your view about the HPV vaccination? (Probes: what is good about it, any concerns about the vaccine or the process of vaccination)

Thank you for your time
• Who have you talked to about having the HPV vaccine? (Probes: parents, friends, school nurse, other health professionals, other family members) What did you discuss?

Experience of giving consent for vaccination
• Who decided whether you would receive the HPV vaccine?
• How were you involved in this decision-making? Were you in agreement with your parents about this decision?

Usability of the e-consent system
• Were you involved in using the e-consent system? If yes, how were you involved either at home or in school and what did you think about this system?
• What did your parents think about the e-consent system? Were there any parts of the online form that they found difficult to complete?
• Were you given the opportunity to find out more about the HPV vaccine via the e-consent system? (Probes: Access leaflets, read and discuss these with your parents) • What do your friends think about the e-consent system?
• How do you think that the e-consent system could be improved?

Acceptability of the e-consent system
• What are your views on giving consent online for your vaccinations? (Probes: any concerns about sharing personal data?) • What did your parents think about this way of obtaining consent?

Experience of the immunisation session
• Could you tell me a bit about your experience of receiving the HPV vaccine in school? What was the session like, how did you feel, what went well, what was difficult?
• How well prepared were you for the immunisation session? (Probe: know you would be vaccinated against HPV, understood what the vaccine was for any why you needed it) • Were you given the opportunity to ask any questions before you were vaccinated? (1) Describe the justification for using a mixed methods approach to the research question.
The use of a mixed methods theory-informed study design is described and justified in the strengths and limitations summary box (page 3, lines 60-62) and the study design section (page 5, lines 112-120). It is also illustrated in Figure 2. The study design allowed us to measure the effect of a pilot e-consent intervention on immunisation performance and identify mechanisms that facilitated or impeded implementation.
(2) Describe the design in terms of the purpose, priority and sequence of methods.
This is described in the study design section (page 5, lines 112-120); the choice of methods was informed by the need to obtain qualitative data to examine the implementation process and quantitative data to assess outcomes and the impact of the intervention.
(3) Describe each method in terms of sampling, data collection and analysis.
These are described in full in the methods section of the manuscript. The sampling frame for the schools participating in the evaluation is detailed in the selection of schools. The boundaries sampling of interviews participants for the qualitative methods were set by this selection. We interviewed all the staff who were actively involved in the implementation process and observed immunisation sessions at schools in each of the boroughs that piloted the intervention. The recruitment and sampling of parent and adolescent interviewees (individual and paired parent/adolescent) is outlined in lines 182-188. We were only able to recruit parents and by proxy adolescents who had ticked a box on the e-consent portal stating they were willing to be interviewed. Informed consent was obtained before all qualitative interviews as described in the ethics section at the end of the manuscript.
(4) Describe where integration has occurred, how it has occurred and who has participated in it. The study team (co-authors) met every 3-4 months to discuss the results of the evaluation and compare quantitative with qualitative findings. The qualitative findings were pivotal for understanding why the pilot intervention did not achieve the expected improvement to consent form return in Year 1 and to identify mechanisms that facilitated or impeded the implementation of the intervention. Collaboration between implementers and evaluators in this real time evaluation also supported the iterative development of the pilot intervention, which is critical to the longer-term integration of new working practices.
(5) Describe any limitation of one method associated with the present of the other method.
I am not completely clear what is meant by this question. The methods complemented each other and were not limited by the presence of another method. The observations informed the interviews and findings from the qualitative data provided insights into implementation challenges which in part explained quantitative findings.
(6) Describe any insights gained from mixing or integrating methods.
As stated in point one the use of diverse methods allowed us to examine the implementation of the pilot intervention and assess its effect on consent form returns (outcomes) and HPV vaccine uptake (impact). Using qualitative methods over a period of a year was very useful to trace how the intervention was adapted over time to improve its acceptability and usability. The design of our evaluation could have been improved if we had been able to repeat the quantitative data collection in Year 2. This was not possible since the new iteration of the intervention was not ready when the original sample of e-consent schools were scheduled to receive the HPV vaccine in Year2. This is a constraint of evaluating interventions in real-time.  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60