Article Text

Identifying the important social outcomes for childhood cancer survivors: an e-Delphi study protocol
  1. Sarah Milner1,2,
  2. Richard G Feltbower1,
  3. Kate Absolom3,
  4. Adam Glaser1,2
  1. 1Leeds Institute for Data Analytics, School of Medicine, University of Leeds, Leeds, UK
  2. 2Leeds Teaching Hospitals NHS Trust, Leeds, UK
  3. 3Leeds Institute of Medical Research, School of Medicine, University of Leeds, Leeds, UK
  1. Correspondence to Dr Sarah Milner; s.milner{at}


Introduction Optimising the health of childhood cancer survivors is important given the high long-term survival rate coupled with a significant late effects burden. Included within the WHO’s definition of ‘Health’ are social outcomes. These are of interest given their impact on adult functioning within society, complex interactions with physical and mental health outcomes and potential for cross generational effects. Categories included within the definition of social outcomes are ill defined leading to potential gaps in research and service provision which could affect the ability of survivors to achieve their maximal potential. An e-Delphi study will be used to achieve expert consensus on the most important social outcomes for childhood cancer survivors to inform future research and ultimately, service provision.

Methods and analysis A heterogeneous sample of at least 48 panel members will be recruited across four groups chosen to provide different perspectives on the childhood cancer journey: childhood cancer survivors, health professionals, social workers and teachers. Purposive sampling from a UK, regional long-term follow-up clinic will be used to recruit a representative sample of survivors. Other panel members will be recruited through local channels and national professional working groups. Opinions regarding breakdown and relevance of categories of social outcome will be collected through 3–5 rounds of questionnaires using an e-Delphi technique. Open ended, 7-point Likert scale and ranking questions will be used. Each round will be analysed collectively and per group to assess inter-rater agreement. Agreement and strength of agreement will be indicated by a median score of 6 or 7 and mean absolute deviation from the median, respectively.

Ethics and dissemination Ethical approval for this study has been granted by Regional Ethics Committee 4, West of Scotland (ID 297344). Study findings will be disseminated to involved stakeholders, published in a peer-reviewed journal and presented at conferences.

  • paediatrics
  • paediatric oncology
  • statistics & research methods

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • Delphi methods will be used to develop consensus on important social outcomes for childhood cancer survivors.

  • A purposive sampling strategy will be employed to recruit patients as members of the Delphi panel.

  • The study will be exclusively online to maximise flexibility for participants.

  • The study is only in English, which may prevent some patients taking part.

  • The methodology from this study may be used as a blueprint to investigate social outcomes in other childhood diseases.


In recent decades, there have been enormous improvements in childhood cancer treatment leading to over 80% 5-year survival in the UK.1 This, alongside an increasing incidence of childhood cancer globally,2 means that more children are becoming adult survivors. There are over 35 000 childhood cancer survivors in the UK3 and 500 000 across Europe.4

It is well recognised that childhood cancer and its treatment leads to long-term health consequences or ‘late effects’. These impact the majority of survivors following treatment, with estimates ranging from 60%5 to nearly 90%,6 increasing with attained age. They encompass physical, psychological and social effects.7 More research is needed to explore long-term psychosocial outcomes in survivors.8

The WHO defines Health as a ‘state of complete physical, mental and social well-being and not merely the absence of disease or infirmity’9 demonstrating the equal importance of positive social outcomes in promoting good health. This definition reinforces the inextricable links between physical, mental and social health and, therefore, the need to study social outcomes in childhood cancer survivors to promote the best health possible in this population.

‘Social outcomes’ is a broad term and because a number of its likely constituents are subjective in nature, it is difficult to accurately categorise. It could include the social determinants of health—‘the conditions in which people are born, grow, live, work and age’10 or ‘a diverse set of social capacities linked to personal functioning and functioning in social structures such as the labour market, social groups and society ’.11 These are helpful definitions and provide a platform to consider ‘social outcomes’, but lack a meaningful breakdown of the domains the term should contain.

Current knowledge

There is not an appreciable body of work to generate a consensus definition, or comprehensive understanding of the term ‘social outcomes’ for childhood cancer survivors.

Current research is weighted towards educational and occupational outcomes. These are recognised as key components of normal social functioning and are easier to quantify.

Multiple studies including systematic reviews have found childhood cancer survivors experience worse educational outcomes than the general population, specifically in those with central nervous system (CNS) disease or following cranial radiotherapy (CRT).12–18 There are higher rates of unemployment, particularly unemployment due to disability and lower net salary among survivors and again, specifically those with CNS disease or following CRT.16 19–25

Childhood cancer can have a negative impact on social relationships including friendships and intimate relationships.26 27 Survivors have lower rates of marriage or cohabitation than the general population28–32 and social isolation may be a problem following treatment.33 Little is known about more nebulous areas such as impact on community life or autonomy.

Other chronic diseases of childhood such as congenital heart disease, type 1 diabetes, asthma and epilepsy similarly lack a clear breakdown of categories of relevant social outcomes yet also exhibit deficits in areas including education, occupation and relationships.34–37

To date, research demonstrates potential deficits in social outcomes experienced by childhood cancer survivors, emphasising a need to better understand which are most important in order to guide future research and, ultimately, finite service provision and patient support.


The objective of this work is to conduct an e-Delphi study to reach expert consensus on the most important social outcomes for childhood cancer survivors.


This study will be conducted using an e-Delphi method. The Delphi method has been used in many health research contexts38–40 to determine the extent to which experts, stakeholders or both agree with each other about a given issue and, via multiple rounds of questionnaires, achieve a consensus opinion.38

There has been some criticism of the lack of clarity around how Delphi processes are conducted,40 highlighting the importance of making methodological processes and protocols openly available.

Given that there is no indication of what the most important social outcomes for childhood cancer survivors are and that this research problem would benefit from collective, subjective judgement, a Delphi study is an ideal technique as it brings together the views of a variety of stakeholders and experts.41 Conducting the study online should improve access and data collection, while reducing the carbon footprint compared with other methodological approaches.

Three to five rounds of e-questionnaires will be administered anonymously to participants via OnlineSurveys ( These will contain suggested categories of social outcomes. The rounds will use 7-point Likert scales to evaluate level of agreement, alongside free text boxes. Between each round, the results will be quantitatively and qualitatively analysed in order to produce the next questionnaire. Resulting data at the end of the rounds will be brought together to devise a consensus list of important social outcomes for childhood cancer survivors. Figure 1 provides an overview of the study.

Figure 1

Flowchart showing overview of the study process. CCLG, Children’s Cancer and Leukaemia Group.

Development of the Delphi round 1 questionnaire

The first-round questionnaire was developed following a scoping review of PubMed, Web of Science and Embase databases for articles using the follow strategy: (child OR childhood OR children OR p?ediatric) AND (cancer OR neoplasm OR tumo?r OR malignancy OR malignant OR leukaemia) AND (survivor) AND (“social outcome*” OR “social health”).

Further searches were performed looking in more detail at areas found during the initial search. Following this, a theoretical set of categories was developed by the research team which encompasses clinicians with expertise in the care of children with cancer and long-term survivorship and health service researchers. A survivor also contributed. These categories are: education, independence and autonomy, work and finances, relationships, community life and lifestyle.

Categories were broken down to increase granularity and seek better understanding of what aspects of each potential category are important. As an example, education is further broken down into questions including the importance of: Receiving any education, completing school up to age 18 (the statutory age to which individuals must remain in education in the UK) and being able to complete higher education.

Patient and public involvement

A childhood cancer survivor contributed to the development of the pilot questionnaire. This underwent cognitive testing with patient and professional groups alongside a lay individual to help identify any ambiguities.42 This led to significant question refinement. It also demonstrated that professional participants were likely to only consider themselves despite being asked to think in the context of the childhood cancer survivor. For this reason, each question will be put to the participants twice- initially for consideration of the importance of the item for the general public and then for survivors. This structured approach was found to better encourage broader consideration of the context.


The study aims to recruit a heterogeneous sample of patients and professionals. The professional group will consist of paediatric oncology doctors and nurses, teachers, and social workers with experience of caring for young people living with and beyond cancer.

There is no set, standard sample size for a Delphi panel. Suggestions include a minimum of 12 panel members in total43 or 10 per area of expertise.44 The study aims to recruit approximately 12 participants per group (12 patients, 12 health professionals, 12 teachers and 12 social workers), and therefore, 48 participants in total. See table 1 for eligibility criteria for all groups.

Table 1

Eligibility criteria for patient and professional samples

If >20 individuals respond for any group during recruitment, it will be regarded as full and all non-responders will be notified. A response rate of 75% per round is required to maintain validity and reduce withdrawal bias.38 Consequently, this will be monitored.

Patient sample

Patients will be recruited to encompass a range of diagnoses as defined by the International Classification of Childhood Cancer, third edition.45 Diagnoses will be grouped into: leukaemias, CNS tumours and other solid tumours.

Potential patient participants will be identified from the long-term follow-up (LTFU) service at Leeds Teaching Hospitals Trust (LTHT), a regional children’s cancer principal treatment centre (PTC) in northern England. The LTFU service comprises approximately 1500 patients of whom 800 are over 18 years old (up to 65 years old).

Assuming up to a 70% rejection/no response rate, invitations will be sent to 40 potential patient participants using a purposive sampling strategy to encourage recruitment of a representative sample of 12 patients. Table 1 displays the eligibility criteria.

Professional sample

The study will recruit health and non-health professionals whose work is relevant to social outcomes: doctors, nurses, teachers and social workers looking after children with cancer (table 1). Doctors and nurses were chosen because of their acute caring responsibilities during diagnosis and their role in longer-term care in which the social effects resulting from treatment become more apparent.

Health professionals will be recruited by surveying members of the Children’s Cancer and Leukaemia Group (CCLG), the professional organisation for individuals caring for young people with cancer across the UK.46 Social workers will be recruited at the regional children’s cancer National Health Service (NHS) PTCs in Leeds and Sheffield (the two main treatment centres for children with cancer in Yorkshire, England) via the social work service for paediatric oncology. Teachers will be recruited through the Leeds Medical Needs Teaching Service and attendees of a local teachers’ workshop which supports and provides information to those teaching children being treated for cancer in Leeds. Recruiting teachers in this way ensures that those involved will have experience of teaching a child or young person with cancer.



Potential patient participants will be identified from clinic lists for the LTFU service at LTHT.

Clinic lists will be obtained for all patients attending medical or nurse-led clinics in person or virtually from 1 January 2021 onwards. Starting at 1 January 2021, the clinical team will identify patients in date order using a purposive sampling strategy (table 2) until a total of 40 patients have been identified. This ensures a range of diagnoses and age groups are approached. The proportion of patients for each disease group are derived from the UK incidence rates for childhood cancers: leukaemias 31%, CNS tumours 25%, other groups including lymphomas 44%.47 The 18–29 years old group make up 50% of the patients as their experiences are likely to be most representative of those coming through treatment into LTFU.

Table 2

Sampling strategy for patient group sample

A letter will be sent to the patient’s registered address by the clinical LTFU team explaining the purpose of the study and inviting them to take part. The team will also note age at diagnosis for future description of the sample. If there is a less than 50% response rate within 14 days, the sampling strategy will be repeated starting at the next patient, in date order from where the previous sample was completed.

If there are insufficient responses using this method, a member of the clinical team will approach patients directly when they attend the LTFU clinic.


Convenience sampling will be used for all professional groups to approach maximum numbers of individuals and increase chances of recruiting the desired number of participants. Clinical professionals: Members of the CCLG will be approached via the Late Effects special interest group. The CCLG will distribute information about the study and invitation to take part on behalf of the study team. Social workers with paediatric oncology experience have been identified in Leeds and Sheffield by their managers. They will distribute email invitations to take part. Similarly, managers from the local Medical Needs Teaching Service will approach their staff, all of whom have experience educating children with cancer. A presentation will be given at the teacher’s workshop inviting attendees to participate.

Letters to patients and emails to professionals will contain a participant information sheet (see online supplemental file 1) detailing the rationale and objectives of the study, what taking part will involve and information on confidentiality and ethical approvals. A variety of methods for registering interest are provided including dedicated email, telephone line and QR code generating an automatic email. Any individual wishing to take part contacts the study team who will provide further information and collect an email address for survey distribution.

Further patients or professionals may be recruited if they present themselves following interaction with those already registered, provided they meet the eligibility criteria.

Study procedure

Recognised key components of the Delphi method41 48 including: (1) use of an ‘expert’ panel, (2) anonymity of panel members, (3) iterative rounds of survey and feedback, (4) controlled feedback at each round and (5) iteration until consensus is achieved will be followed throughout the study. The anonymity component is particularly important in this study as it allows all individuals to be heard equally. Otherwise, for example, there is concern patients may not wish to contradict professionals.

E-consent will be taken at the beginning of the first-round. Involvement in subsequent rounds will be assumed as ongoing consent. Participants will only be able to proceed to the first questionnaire once consent is given.

All participants will be invited to complete each round of the study by email with a link to Online Surveys49 unless they explicitly withdraw from the study or evidence this by not completing a round. If participants withdraw from the study, attempts will be made to ascertain why. Responses from earlier rounds will be kept unless explicitly asked to remove.

Participants will be given 2 weeks to respond to each questionnaire with an initial reminder being sent to non-responders 1 week after the round opens. There will then be 2 weeks for analysis. As a result, each new round will open 4 weeks after the previous round. There will be flexibility in this if low responses prove to be a problem.

Round 1

E-consent, basic demographic information including ethnicity, gender and age alongside participant group will be collected.

Participants will be asked to use a 7-point Likert scale to rate whether listed categories of suggested social outcomes are important for (1) childhood cancer survivors and (2) the general public. This is necessary to ensure that the two groups are considered separately. 7=very important and 1=not at all important. Figure 2 shows an example question as seen on Online Surveys. See online supplemental file 2 for the complete list of round 1 questions.

Figure 2

Screenshot example first round question as it appears on

Free-text boxes will be provided to allow participants to explain their decisions and volunteer other areas they think are relevant but have not been suggested.

Round 2

The responses from round 1 will be collated and analysed. Domains where there is consensus to include or exclude will be identified. Results will be fed back to panel members for comments. Categories not meeting criteria for consensus will be represented to the group alongside the results with the Likert scale as above. Any newly identified categories will be suggested to the group.

Round 3

Responses from round 2 will be collated and analysed. This will again be fed back to the panel with particular attention to areas where consensus has not been achieved. Participants will be asked if they wish to change any opinions based on the analysed feedback from the rest of the group. Participants will then be asked to rank the chosen social outcomes in order of perceived importance from highest to lowest.

If consensus has not been achieved at the end of round 3, up to two further iterations may take place for a maximum of five rounds. At the end of the final round, participants will be notified that no further rounds are required and thanked for their involvement. The study team will undertake final analysis to identify a list of the most important social outcomes for childhood cancer survivors.


All participant groups will be analysed both independently and together. There are no conclusive guidelines for establishing consensus in Delphi literature.38 50

Importance of suggested social outcomes will be measured using the seven point Likert scale. Median scores and strength of agreement will be calculated.

Criteria to include a suggested outcome:

  • Median value≥6.

Criteria to exclude a suggested outcome:

  • Median value of≤2.

Strength of agreement will be assessed using mean absolute deviation from the median. Suggested outcomes not meeting criteria to include or exclude will be represented in round 2 alongside the qualitative feedback (inclusive of any further suggested categories) and median scores for each category. Each score will be calculated overall and according to participant group, age, gender and ethnicity.

To identify themes in free-text question responses, qualitative content analysis51 will be undertaken at each round and presented to participants in the subsequent round. Results will be reviewed by the whole study team to ensure quality is maintained. From round 1, any new categories emerging from the qualitative feedback will be presented alongside categories not achieving consensus.

In the third round, ranking questions will be assessed. Kendall’s W will be used to determine concordance with a coefficient value of 0.7 or greater indicating strong agreement. Inter-rater agreement within and between groups will be assessed using an appropriate statistic such as kappa.

Ethics and dissemination

Ethical approval has been granted by the NHS Health Research Authority’s Regional Ethics Committee 4, West of Scotland (ID 297344). This does not include third party sharing of the data collected. The study will be conducted in accordance with the approved protocol. On completion of the study, results will be fed back to involved stakeholders and then submitted for conference presentation and peer-reviewed journal publication.

Data management

Personal data (names, postal addresses and/or email addresses) will only be collected for the purposes of approaching potential participants, sending out links to the survey and sending out reminders. The names and addresses of potential patient participants will be stored in an encrypted, password-protected file within NHS cloud storage so that they stay within the NHS’s secure data environment. Once potential patient and professional participants have made contact with the study team, their email addresses will be stored in an encrypted, password-protected file within University of Leeds secure cloud storage and data within the NHS will be destroyed.

Data from the study will be downloaded from OnlineSurveys in a pseudonomised format and stored within university cloud storage. On completion of the study, all data will be kept securely for 5 years at the University of Leeds before being destroyed in line with University guidance. Personal data will be destroyed securely 12 months after completion of the study.


Delphi studies are well used in health research, particularly in areas of limited research and/or to explore areas with controversy or lack of clarity. This makes it an ideal methodology for establishing the important social outcomes for childhood cancer survivors. There is significant value in employing the Delphi approach and when used correctly, with rigorous, transparent methodology, it can contribute to increasing knowledge across many areas of health science. However, the lack of clear guidelines around its use, reporting standards, definition of consensus and the numerous modifications to the original method leave it open to criticism.38 41 52 Improved standards of reporting the methodological approach is important so that results can be interpreted with clarity and appropriate caution. Publishing the protocol for this study should help facilitate this.

Choosing the appropriate composition of the panel will strongly affect the results.38 50 52 For this work, the term ‘expert’ was considered broadly with attention given particularly to groups with differing expertise relevant to social outcomes, for example, education. A decision was made to restrict the study to professionals working with children during their treatment rather than those who might only encounter adult survivors. It is hoped that future work will build on this study and include those encountering adult survivors such as within the job market. Including a broader range of healthcare professionals such as occupational therapists would be ideal, but the numbers working specifically within paediatric oncology across the UK are small and it would be impracticable to approach all PTCs and NHS service providers for this study.

Study participation is restricted to English language speakers within the UK. This may risk missing views of some patients within the UK and could limit generalisability across other countries. Limited time and resources available preclude an international study at present. However, it is hoped that this study generates useful information which can be used as a foundation for further work in the UK and other countries. Findings will be shared with national and international groups to stimulate and develop further work in this area.

Key topics missed in the first-round questionnaire are not easily covered in later rounds. There is additional risk of ambiguity and differing interpretation of the questions among panel members which can distort results. In an attempt to combat this, an extensive literature search was undertaken to provide a clear basis for the first-round questions. These were reviewed and refined by individuals meeting eligibility criteria. Space for qualitative comments is built into each question to generate further ideas and identify overlooked areas.

Delphi studies self-select motivated individuals with strong opinions on the topic, particularly over successive rounds, risking bias.50 A sampling strategy is in place in to obtain a representative patient sample. Consideration was given to a further breakdown of the patient group by treatment type, stage at diagnosis and relapse alongside stratifying by index of multiple deprivation but numbers are too small to make this practically possible and they would remain liable to self-selection.

As a questionnaire based study, there is high risk of non-response, increased by the multi-round approach. Keeping participants engaged, feeling they are partners within the study and being persistent with follow-up of non-responders are key to retaining participants.38 50 The recruitment process requires potential participants to contact the study team ensuring that each individual builds a relationship with the team. In addition, a programme of reminders is in place. Keeping the study online, using software which is compatible with mobile devices and allowing survey responses to be saved and returned to later should reduce the study burden and encourage participation.

The data analysis process for a Delphi study is inherently subjective. The whole study team will review all content analysis prior to participant feedback. To further reduce researcher bias, a member check process will be employed following each round of analysis. In this case, a PDF of all qualitative comments will be uploaded to Online Survey for participants to review alongside the new questions and statements.


This paper describes the methodology for a Delphi study to develop expert driven consensus on the most important social outcomes for childhood cancer survivors. It will be the first time this concept has been explored and will help guide subsequent research to maximise the social outcomes for childhood cancer survivors. This methodology will provide a blueprint for social outcomes research in other childhood diseases.

Ethics statements

Patient consent for publication


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • Twitter @DrKateAbsolom

  • Contributors All authors devised the focus of this Delphi study as part of a PhD investigating the social outcomes of childhood cancer survivors. SM is the PhD student and drafted the protocol and manuscript with guidance from AG as lead supervisor and RGF and KA as cosupervisors. SM will recruit participants to the study with support from AG. All authors will contribute to data interpretation, conclusions and final dissemination of results. All authors have read, contributed to and agreed to the final manuscript.

  • Funding This work is part of a PhD studentship supported by a grant from the Candlelighters Trust (registered charity no. 1045077). Grant number: RGEPBI106374.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.