Research question

Among eligible PGs relevant to the assessment, prevention or treatment of common CYMH conditions, which PGs meet criteria for minimum and high quality?

Eligibility criteria

Documents identified using the search strategy described below will be deemed eligible if they meet the following criteria: (1) English language; (2) documents labelled PG, practice parameter or consensus or expert committee recommendations, or documents with the explicit objective or methods to develop original guidance/recommendations; (3) published, revised, updated or reaffirmed between 2005 and 2017; (4) address the assessment, prevention or treatment of one of the following CYMH disorder groups (as defined by Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition),24 mood (major depressive disorder, dysthymia, bipolar disorder), anxiety (agoraphobia, generalised anxiety disorder, social phobia, specific phobia, panic disorder, separation anxiety disorder), self-harm/suicidality, disruptive behaviour (attention-deficit hyperactivity disorder, oppositional defiant disorder, conduct disorder) and substance use disorders and (5) relevant to children and youth ≤18 years of age. Documents meeting the foregoing inclusion criteria will be excluded if judged to be a narrative or systematic literature review that contains summary statements regarding clinical implications/recommendations.

Information sources

Following advice from an experienced health research librarian (MR), we will search Medline, EMBASE, PsycINFO and CINAHL. Our grey literature search will include PG-specific sites (ie, National Guideline Clearinghouse, Canadian Medical Association Infobase, National Institute for Health and Care Excellence, Guidelines International Network International Guideline Library, Australia’s Clinical Practice Guidelines Portal and New Zealand Guidelines Group). It will also target mental health-related organisations (eg, American Academy of Child and Adolescent Psychiatry; Canadian Mental Health Association) and include a broad search using Google. All searches will be supplemented by screening reference lists of eligible PGs, using cited reference searching to reference forward eligible PGs, hand-searching key journals and soliciting recommendations from team members.

Search strategy

Our research librarian will use a strategy that combines subject heading and text terms for mental health AND guidelines AND children/adolescents. The strategy will be developed in Medline and then translated to terms appropriate to other databases and peer reviewed.25 A draft search strategy is provided in online supplementary file 1.

Information management

Search results will be stored using bibliographic management software.

Selection process

One methodologist with expertise in the identification and quality assessment of PGs will independently screen the titles and abstracts of all unduplicated identified records using Reference Manager software. Only documents that clearly do not meet our inclusion criteria will be excluded at this stage, for example, title and/or abstract unambiguously indicates that the document: (1) is specific to adults; (2) does not address one or more of the target CYMH disorders or (3) publication date is prior to 2005. All remaining documents will proceed to full-text screening by two reviewers. A research assistant will obtain full-text records for all potentially relevant documents identified during title and abstract screening. Then two methodologists working independently will apply the inclusion criteria to full-text documents to identify eligible PGs. Disagreements regarding eligibility are then identified and resolved through discussion with the principal investigator (PI).

Data items and collection process

A trained research staff member will extract descriptive data for each eligible PG using a standardised form to capture: date produced; author; organisation type (government, medical society, special group, other); country of origin; CYMH disorder; target population (children and youth only; children, youth and adults); guideline purpose (assessment, prevention, treatment). Training will include refinement of item rewording as needed to improve clarity.

PG quality assessment methods, raters, training

PG quality ratings will be conducted using AGREE II.17 This validated tool is used widely and consists of 23 questions grouped in six domains: scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability, editorial independence. Two additional items assess overall quality. Item response options range from 1 to 7. Two reviewers (MSc in research methods) will participate in a three-stage training exercise: completion of the online AGREE II Overview Tutorial,26 a detailed review of the AGREE II User’s Manual17 and an online practice assessment of an example PG.27 Reviewers will meet with the PI to review disagreements and ‘lessons learned’ about AGREE II. Following training, the two reviewers will independently apply AGREE II criteria to eligible PGs using the My AGREE PLUS online platform.28 For each item, reviewers will indicate their score and justify it by recording document page and paragraph numbers for the information supporting each item in the comment box. When applying AGREE II criteria, reviewers will ensure that any companion documents for a given PG (eg, tools and resources to aid PG implementation, technical reports, health economic analyses, PG evaluation tools, etc, referenced in the main document) were considered in addition to the main PG document. Once the two raters have completed their independent AGREE II ratings, inter-rater differences ≥2 points on initial item scores will be identified and discussed by the two raters and the PI. This cut-point has been chosen as it is both pragmatic and conservative with respect to capturing scoring differences that arise from misinformation (ie, guideline documents are often very long and detailed and it is possible that a reviewer may simply miss important information) rather than differences in judgement regarding the content of the guideline documentation. Following discussion, reviewers are asked to reconsider and possibly revise their item scores; however, numerical agreement on the score assigned for each AGREE II item is not required. Thus, scoring differences that occur due to missed information in the documentation should be resolved during the discussion of disagreements. Any differences that remain following discussion should represent between rater differences in judgement (rather than failure to detect specific pieces of information within PG documents). Final item scores will then be aggregated into six domain scores by summing both reviewers’ scores for all items within a given domain and standardising as a percentage of the maximum possible score (ranging from 0% to 100%) using the formula described in the AGREE II User’s Manual.17

PG high-quality and minimum-quality rating criteria

The AGREE II User’s Manual does not provide criteria to designate PGs as high or low quality. Thus, the interpretation of domain scores and overall PG quality assessments are determined by the user. We will use scores on three AGREE II domains—stakeholder involvement, rigour of development (ie, rigorous consideration of the relevant research evidence base) and editorial independence (management of academic and financial conflict of interest)—to classify PGs according to quality. Minimum-quality PGs will be defined as those that receive a domain score ≥50% on all three domains. High-quality PGs will be defined as those that obtain a domain score ≥70% on all three domains. We selected these three domains because they address the extent to which risk of bias is minimised in the identification and interpretation of the research evidence used to derive the guideline recommendations. The remaining three domains, although important, do not evaluate the clinical validity and trustworthiness of the PG; rather they focus on the problem statement, clarity of presentation and implementability.

Data synthesis and statistical techniques

Research question: How can the quality and usefulness of PGs for CYMH disorders be strengthened?

To develop recommendations to guide future CYMH PG development, we will convene four CYMH PG KU groups, namely clinicians, mental health service planners and youth and adult family members, and engage them in a series of structured exercises derived using nominal group methods and the findings of the SR.31 32

Recommendation Development Group (RDG) membership

Group membership influences the outcomes of consensus exercises. Heterogeneous groups representing the range of relevant perspectives are preferred as they are more likely to produce judgements that reflect a conservative or middle ground.7 33 Accordingly, for each PG user group, we will form groups composed of 10 individuals. Clinicians and mental health service planners will be identified through nominations by project team investigators. Youth (aged 8 to 18 years) and adult family members will be nominated by the members of our youth and adult family member advisory committee and involvement to date in protocol development). Once the four groups are assembled, members will be asked if important viewpoints are not represented, and additional members (up to three) will be invited to participate as needed.

Managing conflict of interest

Methods consistent with current international standards will be used.34 Recommendation Development Group (RDG) participants will be asked to disclose financial or intellectual conflicts relevant to CYMH PG development. The project PI in consultation with other coinvestigators as needed will review all disclosed conflicts and make decisions regarding whether an individual should be recused from all or a portion of the recommendation development exercise.

Generating recommendations

A structured questionnaire, face-to-face meeting interactions and online rating derived from nominal group methods will be used.

Structured premeeting questionnaire

RDG members will respond to an electronic questionnaire prior to a face-to-face meeting. The content of the questionnaire will be derived from the findings of the SR and supported by relevant supplemental information. Provisional questions include:

(1) (A) Using AGREE II as a framework and the SR findings, how should minimum quality standards for CYMH PGs be defined?; (B) Based on the results of the SR, what methodological quality criteria should be the focus of capacity-building initiatives designed to improve PG quality?; (2) What CYMH PGs constitute a core set (including a rationale based on prevalence and burden of illness)?; (3) What content and PG development processes are needed to ensure that PGs are useful to specific types of KUs: (A) clinicians (child psychiatrists, family physicians, paediatricians, nurses, psychologists and social workers); (B) mental health service planners in provincial government ministries, hospitals and community agencies; (C) children and youth and (D) adult family members? and (4) How can increased collaboration between PG development groups be encouraged in order to strengthen PG quality and usefulness, and reduce duplication?

Each question will be linked to supplemental material based on SR findings and/or the content of the minimum-quality and high-quality PGs. For example, for question 1, the quality ratings for each PG (ie, domain scores) will be presented in tabular form. Each RDG participant will be asked to indicate whether each domain is relevant to defining minimum-quality standards, and if so, what score cut-point should be used? Similarly, participants will be asked to define high quality using the AGREE II domains. For provisional question 3 (What content and PG development processes are needed to ensure that PGs are useful to specific types of PG users?), each RDG participant will be provided with electronic access to supplementary materials (eg, template summary of PG content, template summary of PG development process, copy of actual PG) derived from the minimum-quality and high-quality PGs, and asked to identify specific aspects of the content and developmental processes that are essential and/or missing from their user perspective.

Face-to-face meeting

First, the SR findings and collated premeeting questionnaire results will be reviewed in a large group session. Then, each KU group will work independently in small groups with an expert facilitator who is not part of the research team to draft recommendations related to each premeeting question. The goal is to ensure that each PG user group has an equal voice, and that all individuals have the opportunity to express their views. Each small group will then report their draft recommendations for each question to the full group. The principal investigator will collate draft recommendations into a final provisional set for review in the final session of the day. The final set will include draft recommendations that are common to all four PG user groups and draft recommendations that are unique to a specific PG user group.

Generating final recommendations and quantifying the level of consensus

We will finalise the content of the recommendations drafted in the face-to-face meeting and quantify RDG member agreement (0% to 100%) for each recommendation as follows. First, 48 hours after the face-to-face meeting, RDG members will indicate their level of agreement with each draft recommendation using a 7-point scale (1=strongly disagree; 7=strongly agree). Ratings will be collected anonymously in an online survey. RDG members will then be provided with a summary of scores, and participate in a conference call 1 week later to discuss and revise each recommendation as necessary. Finally, RDG members will re-rate their agreement with each revised recommendation in a second anonymous online exercise. Although consensus (ie, 100% agreement among RDG members on the rating assigned to a recommendation) may be achieved for a specific recommendation, this is not our aim. Quantifying the extent of variation in agreement/disagreement with the recommendations produced is an integral part of accurate communication of RDG views.7

Youth and family member participation in protocol development and preparation prior to project start-up

Informed by patient and public involvement methodology, we have already convened a youth and adult family member advisory group (four youth, two adult family members) to provide leadership throughout the project.9 During proposal development, this group first planned the type of involvement they wished to have in the project using the following ‘meaningful engagement’ continuum: consultation, involvement, partnership and shared leadership.35 36 The results of their deliberations are as follows. First, they will participate as learners in the SR to understand PGs and quality appraisal methods. Then, with this preparation, their goal in subsequent project stages will be participation/shared leadership (eg, develop recommendations for how PGs can be more acceptable/useful to youth and adult family members; create dissemination tools tailored to the needs of youth and adult family members). Specific processes and roles are aligned with engagement principles including reciprocal relationships, colearning, partnerships, transparency, honesty and trust.35 36

Training workshops for youth and family members

Four training workshops for youth and adult family members will be conducted prior to participating in the SR, structured nominal group exercises, face-to-face meeting, online survey and conference call. First, youth and adult family members will participate in three half-day workshops to learn about PGs and the AGREE II quality assessment process. The goal is to support capacity development and facilitate their engagement in the SR and recommendation development process. The first workshop will include a 1:1 orientation for each individual and a ‘Terms of Reference’ document will be codeveloped with the entire group. This document will capture the core engagement principals that guide our project and the continuum of meaningful engagement we are using (consultation, involvement, partnership and shared leadership). Workshop 1 will also address processes to support conflict resolution and clinical support.

A fourth workshop will be convened prior to the full team meeting focusing on recommendation development to: (1) prepare youth and adult family members for their role in generating recommendations relevant to PG acceptability/usefulness and (2) enable them to provide input into the full team meeting agenda to ensure a meaningful process that facilitates youth and adult family member engagement in the deliberations and decisions.

We will also conduct briefing sessions prior to and following each workshop, and provide ongoing support as appropriate to facilitate sustained engagement.