Study design

The paediatric appendicitis COS (PA-COS) development will consist of three phases: (1) an update of the 2015 systematic review on outcomes reported in uncomplicated paediatric appendicitis research,13 aiming to identify any additional outcomes used in trials that were published since the previous systematic review; (2) a three-step Delphi study to identify a set of core outcomes from those selected in the literature review. Development of the Delphi is performed according to the checklist by Sinha et al 22 on the design and reporting of Delphi studies concerning COS selection; and (3) an expert panel meeting including physicians, researchers and children/parent representatives in order to ratify the final COS. Children and young people will be involved through their parents during phase 2 and will be engaged directly using a customised approach.

Steering committee

An international steering committee has been established and consists of the following; the authors, a parent/patient representative of the Dutch Foundation: ‘Children and Hospital’ and the lead local investigator of each participating centre (PA-COS development group). The steering committee will agree on the final version of the protocol at the start of the project and will provide input throughout the duration of the project. The steering committee members will also be involved in the development of the final COS. Within the steering committee, a smaller study management group has been appointed which will convene during regular (videoconference) meetings.

Systematic review: treatment outcomes

Hall et al performed a systematic review of RCTs and meta-analyses reporting treatment outcomes of children with appendicitis up to April 2014.13 They reported 115 unique outcomes which were collapsed into a total of 38 standardised outcome terms. We will update the systematic review in order to identify any new unique outcomes in clinical trials or systematic reviews. All RCTs and systematic reviews/meta-analyses reporting treatment outcomes of acute uncomplicated appendicitis in children (<18 years of age) published between 1 January 2014 and 23 November 2017 will be included. The final review will follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline.23 We will search the Cochrane Central Register of Controlled Trials, MEDLINE and EMBASE with the help of a clinical librarian. Additional information on the search strategy/study selection and data extraction can be found in online supplementary S2. Studies reporting only outcomes of treatment in complex or complicated appendicitis (eg, gangrenous or perforated appendicitis, appendiceal mass and appendiceal abscess) will be excluded.

After data extraction, a meeting of the study management group (including NJ Hall) will be held to discuss potential similarities between the outcomes from the 2015 systematic review from Hall et al.13 New unique outcomes will be discussed within the group in order to assign an appropriate standardised outcome term. If these outcomes do not match any of the original 38 outcome terms, a new term will be assigned; this methodology is illustrated in figure 1. The new and original outcome terms will be mapped to four core areas (death, life impact, resource use and pathophysiological manifestations) in accordance with the methods from the OMERACT Filter V.2.0.24 Although Hall et al 13 chose to list the adverse events as a separate core area, we will reclassify these outcome terms to one of the four core areas (table 1). Adverse events of treatment will, however, be labelled separately, as the OMERACT filter suggests.24 A meeting of the study management group will be held to discuss potential similarities between outcomes and to assign appropriate common outcome terms for corresponding outcomes. Outcomes that are found only once and are not generalisable can be excluded (eg, the width of lateral thermal damage of the mesoappendix after appendectomy). Grouping the outcomes under a common outcome term aims to arrive at a manageable and cohesive list of outcomes that is appropriate as a basis for the Delphi questionnaire.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Schematic depiction of outcome term selection from SRs. RCT, randomised controlled trial; SR, systematic review.

Children and young people

This study includes children and young people (5–18 years) who have been treated for acute uncomplicated appendicitis in the preceding 24 months, either with initial NOT or with surgery. Children less than 5 years old are excluded as different outcomes might be appropriate in this very young age group. Also, uncomplicated appendicitis is much less common in young children than in older children. Furthermore, there are no studies in which children below the age of 5 are treated non-operatively. Children will be engaged indirectly as we will urge parents to discuss the answers they provide with their child while filling out the Delphi questionnaire. Young people will be engaged directly through a customised face-to-face approach in selected countries. For the invited children, considering the complexity of the subject and methodology, age is limited to 12–18 years.

Parents

Parents of children and young people (5–18 years) treated for acute uncomplicated appendicitis either with initial NOT or with surgery in the preceding 24 months or during the initial phase of the study were included. Parents will be asked to discuss the answers they provide with their child while filling out the Delphi questionnaire. Parents will be invited to participate by their child’s treating physician or their designate in each participating country/hospital. Participants will be identified retrospectively by contacting patients who were treated in the past 24 months or prospectively by inviting parents to participate after their child has completed the treatment.

Surgeons

General and/or paediatric surgeons who care for children in the specified age group will be asked to participate. Surgeons will be identified and invited by the local coordinators in each participating country. These local coordinators are research groups that have previously registered a clinical trial on uncomplicated appendicitis in children. This should allow for inclusion of physicians who also have experience in research on the treatment of appendicitis.

International online Delphi study

The Delphi method is an effective tool for reaching consensus in a large group without the need for face-to-face contact.26 The use of sequential questionnaires, which are answered anonymously by stakeholders, is an established method for reaching consensus in a group of experts.22 Questionnaires will be sent using DelphiManager,27 a web-based system designed for Delphi studies. The questionnaires will be open simultaneously to all respondents of the participating countries. After each round, the aggregated responses of all participants are shared anonymously in accordance with the Delphi principle.

The list of outcomes from the systematic review will be formatted into questions, accompanied by an extensive plain language summary per outcome, including figures if appropriate. The Delphi questionnaire will originally be formulated in English and will be translated if required. Translation will only be performed by native-speaking professionals.

Participants will be asked to score the importance of each outcome using a 1-point to 9-point Likert scale as recommended by the Grading of Recommendations Assessment, Development and Evaluation working group28 and COMET initiative.18 A score of 7–9 indicates a critical outcome for assessing the effect of a treatment, 4–6 indicates an outcome that is important but not critical and 1–3 indicates an outcome with low importance for assessing the treatment effect. It will also be possible to select an ‘unable to score’ option, which is especially of importance in case parents do not feel equipped to score certain outcomes. The questionnaires, including the plain language summaries, will be piloted by a group of laypersons (n=10) to check for ambiguity and readability.

Delphi round 1

Participants will be divided into two stakeholder groups: parents (with their children) and surgeons. Parents will be asked to discuss the answers they provide with their child while filling out the Delphi questionnaire. Baseline characteristics (age and country) will be ascertained. Parents will be asked if their child was treated with non-operative or operative treatment, time between registration and the first diagnosis of appendicitis, and if their treatment was with or without complications. They will also be asked whether they will be answering the Delphi together with their child. Surgeons will be asked their specialty (paediatric, general, abdominal or other), workplace (academic, teaching hospital or non-teaching hospital), experience with NOT and experience in research regarding appendicitis in children.

All participants will be asked to score all previously identified outcomes according to their perceived importance for assessing the treatment effect. In the first round, there will be an option to suggest additional outcomes not yet listed.

Participants will have between 4 and 8 weeks to complete each round, depending on the response rate. During that time, they will receive a reminder email every 2 weeks as long as they have not replied to the questionnaire.

Delphi round 1: analysis

Results will be analysed by stakeholder group and for all participants using descriptive statistics. Outcomes will be analysed separately for each stakeholder group, as there is evidence that patients are likely to assign importance to outcomes differently from surgeons,29 which has the potential to influence eventual outcome selection.

‘Consensus-in’ will be defined as

• Greater than 70% of participants in both stakeholder groups scoring the outcome as 7–9 and less than 15% in both stakeholder groups scoring the outcome as 1–3.

• Greater than 90% of participants within one stakeholder group scoring the outcome as 7–9. This implies that these outcomes are highly regarded by an individual stakeholder group and should also be included.18

‘Consensus-out’ will be defined as

• Greater than 70% of participants in both stakeholder groups scoring the outcomes as 1–3 and less than 15% of participants in both stakeholder groups scoring the outcome as 7–9. Consensus-out can only be reached when there is consensus across both stakeholder groups.

Outcomes that do not meet any of these criteria will be defined as ‘no consensus’. A stratified analysis will be performed to check for skewing as a result of divergent opinions from a single country, or surgeons with or without research experience.

At the end of round 1, there will be a meeting of the study management group to assess whether an alteration in the Delphi study is appropriate. If additional outcomes are suggested by Delphi participants, each outcome will be assessed by the study management group to determine whether it is indeed new and to which category it should be classified. Wording of the Delphi questionnaire will be adjusted if misinterpretation is suspected.

Delphi rounds 2 and 3

All participants who complete the previous round will be asked to participate in the next round. Only outcomes that have not yet been defined as consensus-in or consensus-out during the previous round will be presented in the following rounds to all participants. Outcomes for which there was only consensus-in within a single stakeholder group will still be presented to the other stakeholder group to evaluate whether consensus can be achieved in both stakeholder groups. An overview of included and excluded outcomes will be available. The outcomes for which there is no consensus and the newly suggested outcomes from the previous rounds will be presented with the participants’ individual scores and the median scores from each stakeholder group combined with a histogram showing the scoring distribution. Participants will be asked to score all remaining outcomes in the same manner as in round 1.

Delphi rounds 2 and 3 analysis

Results will be analysed per stakeholder group and for all participants using descriptive statistics, including a stratified analysis. The same definitions for consensus in/out as in the first Delphi round are upheld. After the second round, there will be a meeting of the study management group to assess the need for alterations in the Delphi study and to decide whether or not to proceed with a third Delphi round, assuming consensus between both stakeholder groups on more than 80% of the outcomes and more than five outcomes with consensus in. To give an estimate of the degree of agreement between respondents, the width of the IQR of the median ranking score will be calculated, potentially ranging from 0, meaning complete agreement, to 8, meaning least possible agreement. This will be calculated for both the individual stakeholder groups and the entire group of respondents after the final round.

Ethics

The medical research ethics committee of the Academic Medical Center Amsterdam confirmed that the Dutch Medical Research Involving Human Subjects Act does not apply to this study and that complete approval of this study by the committee is not required. Each participating country/research group will be asked to obtain ethics board approval or confirm that ethics board approval is not required. Electronic informed consent will be obtained from all participants. The face-to-face engagement of young people (12–18 years) will take place in selected countries, and separate ethics board approval will be obtained, as appropriate.

Data collection and confidentiality

All data will be handled confidentially and in accordance with the Dutch Personal Data Protection Act and the European General Data Protection Regulation. DelphiManager27 will be used for the online questionnaire. After obtaining informed consent from all participants, only limited identifying information (name and email) will be ascertained during registration. This information will be stored separately from the answers given in the questionnaire and will be used only for the purpose of direct feedback and reminder emails. Access to personally identifiable data will be strictly limited.

Study status and dissemination

In the first quarter (Q1) of 2018, the following 13 countries were invited to participate in the project: the Netherlands, USA, Canada, Australia, Sweden, Finland, UK, France, Italy, Israel, Japan, Singapore and Malaysia. Ten countries replied; Italy, Israel and Japan did not. In Q1 of 2018, the systematic review was finished. In the second quarter of 2018, the Delphi questionnaire was developed and piloted. In the third quarter of 2018, all materials were translated. Between the fourth quarter of 2018 and Q1 of 2019, institutional review board applications were submitted in 10 countries and 15 participating centres. The anticipated start of the online Delphi study is May 2019. We anticipate to have the final COS ready by Q1 of 2020. Dissemination of the results will be accomplished by publication in an international peer-reviewed scientific journal and by presentations at (international) conferences. By involving the majority of the principal investigators who are currently involved in research on uncomplicated appendicitis in children, we aim to optimise uptake of the final COS. By involving journal editors and healthcare commissioners in the face-to-face consensus discussion, we aim to ultimately have the COS introduced as a requirement in future outcome reporting on the treatment of uncomplicated appendicitis in children. We will also actively send out the final COS to relevant journal editors and funding bodies to promote uptake in future research.

Outcomes selection

The selection of potential outcomes will be done systematically and will provide a selection for the first Delphi questionnaire that reflects most issues pertinent to the treatment of uncomplicated appendicitis. By including systematic reviews/meta-analyses that also report on non-comparative studies, we expect to identify all reported treatment outcomes, including those from the relatively new field of NOT for uncomplicated appendicitis.

To be able to arrive at a manageable list of outcomes that is appropriate for a Delphi study, the number of outcome terms needs to be somewhat limited. In order to achieve this, the outcomes derived from our systematic review will be merged in case of similarity. If outcomes are not generalisable and are reported only once, they will be excluded. This will be proposed and prepared by two independent reviewers and discussed in the study management group. However, the merging of outcomes will inevitably lead to some loss of detail.

Global consensus

In order to reflect the views of different stakeholders, a variety of groups will be part of the development of this COS. This is the case not only on a national level but also on an international level, related to, for example, differences between countries in resources, treatment practices for acute uncomplicated appendicitis and cultural differences. For example, there is a large difference with regard to the standard length of hospital stay after an appendectomy for uncomplicated appendicitis. In the USA, much effort is devoted to reduce the number of admission days; in the UK, there is only limited attention for the duration of admission, and for instance, in Japan, an admission for 5 days is not uncommon. We can expect that these kinds of differences result in different opinions regarding the COS. By also involving patients and parents from the participating countries, we hope to correct for these differences.31 In conjunction with the UK PA-COS research group, we decided that an international validation of the UK COS would not give the depth of information and would not allow for consensus formation on all possible outcomes, which we feel is appropriate, considering the previously mentioned significant differences between countries. Involving members from different countries should lead not only to the development of a COS that reflects the opinions of the international community but also to an internationally applicable ‘minimal’ COS. However, selecting the participating countries on the basis of their involvement in research on appendicitis in children is a limitation. This choice was made on the basis of feasibility. Researchers in the field of uncomplicated appendicitis have an interest in the development of a COS and have the network to help carry out the Delphi study. With our current selection, we will still have participants from four different continents. Our method of country selection has another advantage. Since NOT is an important research subject in childhood appendicitis, we aim to include surgeons and parents who have experience in that field. As NOT is still experimental in most countries, we also need surgeons and patients who have been involved in such research.

Limited face-to-face consensus

If consensus is reached in the Delphi study, we will not be organising a formal consensus meeting. The Delphi method can be used for reaching consensus in a group of respondents without the need for face-to-face contact. There is a risk of bias if a face-to-face consensus meeting leads to selection of only participants who are able to attend the meeting, which is especially a problem in a global consensus procedure. There are also problems regarding language barriers in an international consensus meeting. To check for interpretation errors in the Delphi method and to ensure a pragmatic and well-defined set of outcomes, the results of the Delphi study will be discussed in an (international) expert meeting. However, the influence of this meeting on which outcomes are selected for the final COS is very limited, as this selection is primarily made in the Delphi study.

Involving parents and their children

Involving patients in COS development has recently become common practice with 88% (n=112 as of 12 April 2016) of ongoing COS development studies doing so.18 Involving patients as participants seems imperative as patients may select different outcomes, compared with physicians.16 For this protocol, we performed a scoping review (unpublished data, Knaapen M, 2018) that found 12 studies that directly engaged children in COS development, either as part of the advisory group or the steering committee, or as a stakeholder group in the Delphi,15 22 focus groups,32 interviews33 or as a part of the consensus meeting.34 Attempts to engage children and young people in an online Delphi questionnaire have proven to be difficult. In the UK COS for uncomplicated appendicitis, there were substantial difficulties with retaining young people in the consecutive rounds of the Delphi questionnaire, despite extensive efforts to optimise the methodology to appeal to children and young people, including preliminary semistructured interviews on the subject, pretesting of the Delphi survey by young people and children,15 and video animations explaining the need for a COS. However, parent participation showed more promising results. Consequently, to safeguard the input of children/young people, the Delphi questionnaire for this study will be developed to be completed by parents with input from their children (5–18 years) whenever possible. In order to ensure that there are no large discrepancies between the opinions of parents together with their child, and with children without their parents, we will organise a form of in-person interaction with young people (12–18 years) who have been treated for appendicitis. Involving children/young people in COS development is a subject of interest in many ongoing COS development projects. As the search for the optimal approach to engage young people is ongoing, we have not yet selected a final methodology. Two members of the study management group are currently involved in a group that is developing such methodology in consultation with young people themselves. We will update our protocol as soon as we settle on a methodology before starting the face-to-face engagement. The updated protocol will be published on an online, open-source format (via the Open Science Framework).

A limitation is that due to the international nature of our study, it will not be feasible to engage children directly in all the participating countries. That is why the face-to-face engagement will take place in selected countries.

Other stakeholders

After careful consideration and consultation with the participating countries, it was decided not to include paediatricians, general practitioners, nurses or emergency medicine physicians. Although all these specialists play an intricate role in the diagnosis and care for children with appendicitis, they do not make the final decision regarding treatment or its provision. However, we will however, depending on the organisation of the healthcare system in each country, ask these stakeholders to comment on the final COS in order to ensure that essential outcomes are not missed. Since almost all research regarding treatment of paediatric uncomplicated appendicitis is initiated by (paediatric) surgeons, it was decided that researchers will not be included as a separate individual stakeholder group. However, involvement in research will be registered. While their opinion is vital to the development of a COS, it is likely researchers will be well represented in the (paediatric) surgeon stakeholder group. A stratified analysis will be performed to check for skewing of the results by surgeons involved in research. It was also decided not to include journal editors or healthcare commissioners. Even though their opinion is of great importance especially regarding implementation, it was determined that their opinion is not essential in establishing the outcomes selected for the COS. Also, there is much variability between countries regarding the role of these stakeholders, which would lead to major challenges regarding Delphi analyses of such a small stakeholder group. However, to enhance implementation and because of their expertise on the use of COSs, representatives of these stakeholder groups will be asked to attend the final consensus discussion.

Outcome measures

This study will not answer the question on how to measure the outcomes that are included in the final COS or at what time point the outcomes should be measured. However, we will attempt to come to a clear definition of each outcome. We expect that further research will be necessary to answer the question of timing and how to measure the outcomes. We will advise on this subject in the final report.