Article Text

Download PDFPDF

STANDING Collaboration: a study protocol for developing clinical standards
  1. Louise K Wiles1,2,
  2. Peter D Hibbert1,2,3,4,5,
  3. Jacqueline H Stephens1,
  4. Enrico Coiera5,
  5. Johanna Westbrook3,
  6. Jeffrey Braithwaite2,
  7. Ric O Day6,
  8. Ken M Hillman7,8,
  9. William B Runciman1,4
  1. 1 Patient Safety and Healthcare Human Factors, Centre for Population Health Research, School of Health Sciences, Sansom Institute for Health Research, University of South Australia, Adelaide, South Australia, Australia
  2. 2 Centre for Healthcare Resilience and Implementation Science, Australian Institute of Health Innovation, Faculty of Medicine and Health Sciences, Macquarie University, Sydney, New South Wales, Australia
  3. 3 Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Faculty of Medicine and Health Sciences, Macquarie University, Sydney, New South Wales, Australia
  4. 4 Australian Patient Safety Foundation, Adelaide, South Australia, Australia
  5. 5 Centre for Health Informatics, Australian Institute of Health Innovation, Faculty of Medicine and Health Sciences, Macquarie University, Sydney, New South Wales, Australia
  6. 6 St Vincent’s Clinical School, University of New South Wales, Kensington, New South Wales, Australia
  7. 7 The Simpson Centre for Health Services Research, South Western Sydney Clinical School, The University of New South Wales, Kensington, New South Wales, Australia
  8. 8 Intensive Care Unit, Liverpool Hospital, Liverpool, New South Wales, Australia
  1. Correspondence to Professor William B Runciman; william.runciman{at}


Introduction Despite widespread availability of clinical practice guidelines (CPGs), considerable gaps continue between the care that is recommended (‘appropriate care’) and the care provided. Problems with current CPGs are commonly cited as barriers to providing ’appropriate care'.

Our study aims to develop and test an alternative method to keep CPGs accessible and up to date. This method aims to mitigate existing problems by using a single process to develop clinical standards (embodied in clinical indicators) collaboratively with researchers, healthcare professionals, patients and consumers. A transparent and inclusive online curated (purpose-designed, custom-built, wiki-type) system will use an ongoing and iterative documentation process to facilitate synthesis of up-to-date information and make available its provenance. All participants are required to declare conflicts of interest. This protocol describes three phases: engagement of relevant stakeholders; design of a process to develop clinical standards (embodied in indicators) for ‘appropriate care’ for common medical conditions; and evaluation of our processes, products and feasibility.

Methods and analysis A modified e-Delphi process will be used to gain consensus on ‘appropriate care’ for a range of common medical conditions. Clinical standards and indicators will be developed through searches of national and international guidelines, and formulated with explicit criteria for inclusion, exclusion, time frame and setting. Healthcare professionals and consumers will review the indicators via the wiki-based modified e-Delphi process. Reviewers will declare conflicts of interest which will be recorded and managed according to an established protocol. The provenance of all indicators and suggestions included or excluded will be logged from indicator inception to finalisation. A mixed-methods formative evaluation of our research methodology will be undertaken.

Ethics and dissemination Human Research Ethics Committee approval has been received from the University of South Australia. We will submit the results of the study to relevant journals and offer national and international presentations.

  • quality indicators
  • healthcare
  • practice guidelines as topic
  • consensus development conferences as topic
  • quality assurance

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • We will develop and evaluate a method for generating and ratifying clinical standards and indicators of ‘appropriate care’ for common health conditions which has been designed to overcome deficiencies in current methods.

  • This study will obtain expert consensus on ‘appropriate care,’ underpinned by evidence, for a range of common medical conditions.

  • The recruitment of healthcare professionals, patients and consumers to review clinical indicators may introduce selection biases.

  • The use of English language clinical practice guidelines may not be representative of all available evidence, and limits the generalisability of study findings.


In Australia, ‘appropriate care’ (care in line with evidence or consensus-based guidelines) is provided to adults, on average, only 57% of the time, with large variations across common medical conditions and providers.1 Problems with clinical practice guidelines (CPGs), standards and indicators (see box 1 for definitions) are commonly cited as one of the barriers to providing appropriate care2; these include large numbers of repositories and guidelines; duplication and overlap among guidelines; differing recommendations for care practices; lack of currency; inconsistent structure and content; voluminous documents which are not easy to assimilate or use3–5; and recommendations which are often vague and difficult to measure.2 6 7 In addition, most CPGs lack detail of how evidence was interpreted and weighted to formulate recommendations, offer little opportunity for end-users to provide formal feedback8 and have been developed by people with (often undisclosed) professional or commercial conflicts of interest (COIs).2 6 9

Box 1

Definitions for clinical practice guideline, standard, indicator and tool.

A clinical practice guideline:

‘Statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.10 11

A clinical standard2:

  • is an agreed process that should be undertaken or an outcome that should be achieved for a particular circumstance, symptom, sign or diagnosis (or a defined combination of these)

  • should be evidence based, specific, feasible to apply, easy and unambiguous to measure, and produce a clinical benefit and/or improve the safety and/or quality of care, at least at the population level.

If a standard cannot or should not be complied with, the reason/s should be briefly stated.

 A clinical indicator2:

  • describes a measurable component of the standard, with explicit criteria for inclusion, exclusion, time frame and setting.

A clinical tool2 6 12–14:

  • should implicitly or explicitly incorporate a standard or a component of a standard

  • should constitute a guide to care that facilitates compliance with the standard

  • should be easy to audit, preferably electronically, to provide feedback

  • should be able to be incorporated into workflows and medical records.

Implementability of CPGs is a key factor affecting their perceived utility and uptake11 15; in response, international efforts are being directed at developing clinical standards and indicators to identify evidence and service delivery gaps and areas for improvement, and understand and measure the quality of care provided.16 17 Emerging schools of thought suggest that ‘appropriate care’ may be enhanced through greater patient (health consumer) engagement.12–14 18 This could be facilitated by involving patients and interested laypeople as well as healthcare professionals (HCPs) in CPG development,6 19 20 and using online technologies to enhance transparency, accessibility and currency of both content and development processes.6 21

The strategies employed in this protocol aim to mitigate problems with existing CPG development processes (table 1) by adopting a single approach to avoid duplication, using an ongoing and iterative documentation process to facilitate transparent synthesis of up-to-date information and make its provenance accessible, and requiring all participants to declare their COIs.2 Indicators will be developed using selection criteria to reflect ‘essential’ clinical practice and be expressed one concept at a time in plain English to help create standards that are clear, concise, measurable and easy to use.

Table 1

Key aspects of the STANDING Collaboration methodology, designed to mitigate issues with current CPG development

The aims of the STANDING Collaboration study are to (1) provide proof of concept for an alternative method for creating sets of nationally-agreed evidence-based standards and clinical indicators, and (2) obtain consensus on ‘appropriate care’ for a range of common medical conditions. To do so, we will use a three-phase approach to engage relevant stakeholders, develop clinical indicators representative of ‘appropriate care’ (which constitute the standard) for a range of common conditions, and evaluate our processes, products and feasibility. We plan to develop an inclusive, transparent, collaborative process, which allows HCPs and patients or consumers to develop and keep up-to-date clinical standards comprising indicators with defined attributes, using an online curated wiki-based platform to facilitate ongoing review and updating of the standard, or individual indicators, as soon as new evidence emerges. In this study, the term ‘wiki’ refers to an interactive information management system which will allow users (eg, HCPs and patients) to collaborate directly in formulating and refining indicators that are relevant to their clinical practice and lived experience.21 22 The source and provenance of each indicator, including all suggestions, will be posted online and updated as necessary.

Methods and analysis

Our three-phase approach (figure 1) comprises:

Figure 1

Overview of STANDING Collaboration research methodology and definitions for participant groups. CPG, clinical practice guideline.

  1. stakeholder analysis;

  2. development and test of a process for creating clinical indicators representative of ‘appropriate care’ for a range of common conditions;

  3. evaluation of processes, products and feasibility.

Phase 1

In order to gain an understanding of potential barriers, facilitators and the overall feasibility of the STANDING Collaboration methodology, stakeholder perspectives will be captured through a series of semistructured qualitative telephone interviews. HCPs and consumers will be invited to participate. Relevant medical colleges, professional and consumer associations and networks will be contacted, using publicly available information, to request assistance with the recruitment of interview participants. Invitations will comprise email notifications to members and media releases and articles within newsletters, asking potential participants to contact the Research Team members. The telephone interviews will be conducted at a time convenient to the research team member and the participant. Based on the sample sizes reported in similar stakeholder analyses,23 24 we anticipate conducting approximately 18–25 interviews in total (9–13 interview participants per stakeholder type), or until saturation is reached.

Using guidelines for stakeholder analyses25 and the schedules from previous qualitative research as a guide,19 24 a range of topics will be explored in the interviews (table 2).

Table 2

Topics (and their rationale) for exploration in the qualitative stakeholder interviews

Interviews will be recorded with consent and transcribed by a professional transcription company. Transcripts and summaries will be returned to participants for the purpose of making comments or corrections and providing feedback on the findings.26 Content analyses will be used to derive common themes. We will use open coding and inductive reasoning with two coders to group similar responses into categories and assign labels capturing specific themes. Any discrepancies will be discussed among Research Team members. After agreement is reached on the composition and label for each category, we will assign (axial coding) categories to the central phenomena of interest (table 2).27 28

Phase 2

Clinical indicators will be developed for individual conditions using a four-stage process:

  1. source, select and search relevant CPGs;

  2. extract all concepts from each CPG together with the relevant text in which they appear (original recommendation), and tabulate common concepts to select, draft and format the proposed clinical indicators based on identified concepts;

  3. review the indicators internally;

  4. review the indicators externally.

Stage I: source, select and search relevant CPGs

Interview data from the stakeholder analysis (phase 1) will be used in conjunction with national health priority areas, burden of disease and prevalence data to identify candidate conditions for clinical standard and indicator development.1 7 Clinical indicators will be drawn initially from the latest CPGs. A systematic search will be undertaken of national-level Australian CPGs endorsed by the National Health and Medical Research Council, and international-level guidelines from the National Institute for Health and Care Excellence in England, Scottish Intercollegiate Guidelines Network, and the Agency for Healthcare Research and Quality’s National Guideline Clearinghouse in the USA, and the Guidelines International Network. In the absence of Australian national or international CPGs, relevant professional medical college and association CPGs may also be searched, as well as those published at state or professional level and in international journals. Details of the search strategy are provided in appendix A, online supplementary file 1.29 30 In order to describe the quality of the evidence sources from which our clinical indicators will be derived, two members of the Research Team will independently appraise CPGs using the AGREE-II (Appraisal of Guidelines for Research and Evaluation II) tool.31–34

supplementary appendix A

Stage II: select, draft and format proposed indicators

Recommendations (and their key underlying concepts) from each CPG will be collated and used to inform the content of the proposed clinical indicators. Not all recommendations published in CPGs will become indicators. Recommendations will be flagged for potential exclusion based on the following criteria:

  • strength of the wording of the recommendation (ie, ’may' and ’could' statements would be excluded; ’should' and ’must' statements would be included)

  • vague guiding or aspirational statements and those without recommended actions

  • conflicting recommendations from less recent CPGs and those with lower AGREE-II scores.

All clinical indicators will be written in plain English, one concept at a time, using a structured and standardised format (eg, commencing with the inclusion criteria followed by the compliance action)7 (table 3). For each condition, indicators will be arranged according to phases of care (ie, screening, diagnosis, assessment, acute care, ongoing care) so that together, they constitute a clinical standard amendable to inclusion as a clinical tool over the patient journey.

Table 3

Examples of clinical indicators from CareTrack Australia 1 that were written in plain English, one concept at a time using a structured and standardised format

Stages III and IV: iterative review using a modified e-Delphi approach

Stages III and IV will involve online wiki-based reviews of proposed clinical indicators; initially by way of an internal review between the Research Team and Curator Group (stage III), followed by an external review by the Curator Group and HCPs and consumers who choose to register to the site (Wiki Registrants) and provide feedback on the proposed clinical indicators.7 This approach has been chosen to facilitate the number and spread of individuals’ perspectives obtained and avoid domination of the consensus process by one or a few participants, and to optimise face validity of the final set of clinical standards and indicators.7 35 All STANDING Collaboration participants (Research Team, Clinical Champion, Curator Group members, and Wiki Registrants) will be required to complete a COI declaration, which will be taken into consideration when accepting or rejecting suggestions, and logged with the provenance of each indicator for transparency.9 36

Stage III: internal review processes

Internal reviews will first be conducted within our Research Team, and subsequently by our Clinical Champion and Curator Group members who will comprise a mix of at least two members of the following: clinicians (eg, general practitioners, medical specialists, allied health professionals, nurses), researchers, policymakers or public health specialists or healthcare quality improvement experts, and consumers. Depending on their self-reported scope of practice, expertise and interest, Curator Group members (including consumers) may be able to participate in review panels for more than one condition. The total number of invitations to potential Curator Group members will depend on the skill mix of those invited and the overall pattern of recruitment. This selection strategy is supported within the Delphi process literature with studies using similar criteria to choose potential participants (eg, renown, member of an organisation, recommendation, years of experience, willingness to participate, availability, interest).7 35

The internal review will consist of a maximum of three rounds to allow sufficient testing of the subsequent phase of the process (ie, external review). In the first round, drafts of proposed clinical indicators, and the recommendations on which they are based, will be sent via email to the Curator Group members. The review criteria to be used are based on the methods from previous studies for developing and measuring indicators of appropriate care.1 7 16 37–39 Curator Group members will be asked to: recommend indicators for inclusion (with or without amendments) or exclusion, provide comments in relation to three key criteria: evidence, feasibility and importance (appendix B, online supplementary file 2) and make additional suggestions (with supporting material). In addition, Research Team members will pose specific questions to the Curator Group members about individual indicators to highlight inconclusive or conflicting CPG recommendations, or to clarify definitions for inclusion criteria and compliance actions. In particular, consumer members of the Curator Group will be asked to vet the plain English wording of clinical indicators and a linked glossary of terms to ensure that content is appropriately targeted to the consumer audience. In this round, Curator Group members will complete their assignments independently to minimise ‘group-think.’40 41 Research Team members (WBR, PDH, LKW, JHS) will collate the feedback and revise the content, structure and format of each indicator. The refined set of indicators (including the original indicators and any feedback and suggestions) will be sent to the same Curator Group members for a second round of scoring. The same approach will be used in the second round, with a request for further refinement and identification of indicators to be included or excluded. If necessary, Curator Group members will discuss the proposed set of indicators via a third round teleconference, with a view to achieving consensus and approving the indicators for the external online wiki-based review process.

supplementary appendix B

Stage IV: external review processes

External reviews will be conducted by HCPs and consumers who have registered to this wiki as reviewers (Wiki Registrants). Relevant medical colleges, professional and consumer associations and networks will be contacted to request assistance with the identification of potential clinical indicator reviewers. Invitations will be by email, media releases and articles within newsletters. HCPs and consumers will self-nominate as reviewers for one or more of the STANDING Collaboration conditions based on their interests, scope of practice and experience.35 42 Wiki Registrants for this process will be required to declare their COIs, which will be taken into account by the Clinical Champion and Curator Group when considering reviewers’ feedback on the indicators.9 36

The external review will involve an interactive wiki-based process where indicators for each condition from round 3 of the internal review will be posted on an online wiki site. A software development company will be engaged to purpose design and custom build the wiki for this project. The wiki ‘live’ time for each version will depend on the recruitment rate of reviewers and the progress of their reviews, but is anticipated to be no longer than 3 months per round. Reviewers will provide comments on indicators in relation to the three key criteria: evidence, feasibility and importance (appendix B, online supplementary file 2), make recommendations (ie, inclusion, inclusion with amendments, exclusion, hold) and be able to suggest edits in real time. The Clinical Champion and Curator Group for each condition will follow-up and manage external reviewers’ responses, and make final recommendations for that version regarding the inclusion, content, structure and format of indicators. The Clinical Champion and Curator Group will use supporting references when considering and responding to each suggestion related to whether and why they have been included or rejected. In addition, all external reviewers’ comments and recommendations will be logged, classified and presented in subsequent rounds according to whether and why they have or have not been incorporated into the next iteration. This will allow tracking of the evolution of the standards and indicators from the original recommendations on which they were based to their final iteration, as well as the nature and influence of review feedback in shaping the standard. Once the indicators are ‘stable’ with no further significant changes being suggested, that version of the standard will be published as comprising a set of clinical indicators that represents ’appropriate care' for Australians with the candidate conditions at that time. Endorsement will then be sought by relevant professional bodies and consumer organisations. For each medical condition that has undergone indicator development via the STANDING Collaboration process, it will be possible for evidence to be monitored by the Curator Group (or a subgroup comprised of key members of the Curator Group) in order to update standards and indicators as necessary. For each condition, our initial monitoring plan involves using information from automated database searches and feedback from the wiki to initially update indicators every three months and, once stable, at a minimum of every six months.

Phase 3

A multimethods evaluation of the process and products of phases 1 and 2 of our research methodology will be undertaken (table 4). Three data sources will be used to inform the evaluation: (A) engagement and utilisation statistics sourced from the wiki logs—these will include demographics of users and rates, times and nature of use; (B) the nature and content of Wiki Registrant, Curator Group and Clinical Champion comments (eg, the format and rationale of proposed changes to indicators, level of agreement between reviewers and resulting changes to the indicators); and (C) all users and stakeholders’ perspectives on the process, usability and appropriateness of the vehicle for developing the standards (ie, the wiki) as well as the acceptance and utility of the final sets of standards and indicators themselves.

Table 4

Description of the phase 3 evaluation of phase 1 and phase 2

Participants will be invited to provide feedback regarding their experiences and perspectives via one of three data collection methods: (1) online user perspectives survey (for external reviewers); (2) semistructured interviews (for Review Panel members and the Clinical Champion); and (3) interviews and focus groups (for phase 1 stakeholders, following publication of the sets of indicators) (table 4). Both quantitative and qualitative analyses of these three data sources will be undertaken including: descriptive statistics for the characteristics of wiki users, their engagement and patterns of use, frequency counts and content analyses of ratings and free-text responses from the online user perspectives survey, semistructured interviews and focus groups (table 4). Using these data, recommendations will be developed regarding the overall feasibility of the wiki process for future indicator development. The next phase of the research is to study the implementation of the standards as a translational tool for clinical practice and ongoing audit.

Ethics and dissemination

Ethical approval

Human Research Ethics Committee approval has been granted from the University of South Australia (protocol number 0000035183). All STANDING Collaboration participants and reviewers will be required to give informed consent (for their chosen study phase(s)) and complete a COI declaration prior to participation, which will be recorded and managed according to an established protocol (appendix c, online supplementary file 3).43–46

supplementary appendix C


We will submit the results of the study to relevant national and international journals with the intention of publishing the results widely. As well, we will make national and international presentations to stakeholder groups including those involving patients, researchers, clinicians, managers and policymakers.


Notwithstanding the large number of CPGs currently available, delivery of ‘appropriate’ healthcare in Australia and internationally is highly variable and leaves considerable room for improvement. A number of major difficulties have been identified with current CPGs and their development. Our alternative approach for keeping evidence accessible and up to date has been designed to mitigate problems with existing processes. Here, we describe a protocol for developing and testing a process for creating clinical standards, embodied in clinical indicators.

This process has been designed to systematically address many of the problems identified with current CPGs and their development. The approach is characterised by being inclusive (HCPs, researchers and consumers), transparent (all reviewers’ suggestions are logged, with their provenance, as accepted or rejected by the Clinical Champion and Curator Group), up to date (revisions will be ongoing after a version is published), easy to use (one concept per indicator), written in plain English and able to be integrated into the sequence of work flow in managing a condition.2 29 30

Findings from this study will be used to inform the design of future studies using Delphi processes to establish consensus on recommended healthcare, and will be relevant for national and international researchers, policymakers, healthcare practitioners and patients. Specifically, there is potential for standards and indicators developed using this methodology to be assembled to comprise the content of electronic tools for the basic care of common conditions (ie, reflects ‘essential’ Australian clinical practice; appendix B, online supplementary file 2). It is envisaged that the clinical tool (box 1) will2:

  1. implicitly or explicitly enunciate the clinical standard for the basic care of the condition in question

  2. inform HCPs, patients and carers about that condition

  3. guide care

  4. document what care has been offered and what has not (and why)

  5. be amenable to audit (preferably automated) so that feedback can be provided—at clinical indicator, patient, provider, facility and eventually population levels.

We recognise several limitations to our study. The inclusion of only English language CPGs, for pragmatic purposes and contextual consistency, may not be representative of all available evidence and limits the generalisability of our findings. Reviewers will be invited to participate and self-nominate for conditions that are within their scope of practice, interest or experience (ie, healthcare providers and consumers) which introduces a selection bias. Clinical indicators will be developed from recommendations in existing CPGs, which means there is potential for problems with current CPG development processes to contaminate our final sets of indicators and standards. The aim of this study is to provide proof of concept and test a new methodology. Our intention is that the key approaches and characteristics of the STANDING Collaboration clinical standard and indicator development process (ie, governance structure, transparency, HCP and consumer engagement and codesign, access to provenance of both accepted and rejected suggestions, and use of online technologies to facilitate keeping the indicators up to date) will be universally applicable, and be able to be tailored to other healthcare settings and structures.30 47 48 We aim to ameliorate these limitations by adopting a collaborative user-centred approach where feedback from both consumer and HCP groups is sought, communicated transparently and incorporated into guidance for others who wish to develop clinical standards and indicators.6


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.


  • Contributors WBR is the chief investigator and together with PDH, EC, JW, JB, ROD and KMH initiated the project and led the NHMRC grant proposal. LKW is a Research Team member and developed the first drafting of the protocol manuscript. LKW, PDH, JHS and WBR iteratively developed the methodology. All authors (LKW, PDH, JHS, EC, JW, JB, ROD, KMH, WBR) contributed to the writing of this manuscript.

  • Funding This work is supported by NHMRC Program Grant APP1054146. It is led by the University of South Australia and the Australian Institute of Health Innovation, Macquarie University.

  • Competing interests None declared.

  • Ethics approval University of South Australia Human Research Ethics Committee.

  • Provenance and peer review Not commissioned; externally peer reviewed.