Article Text

PDF

A survey of radiation treatment planning peer-review activities in a provincial radiation oncology programme: current practice and future directions
  1. Michael Brundage1,2,
  2. Sophie Foxcroft2,3,
  3. Tom McGowan4,5,
  4. Eric Gutierrez2,
  5. Michael Sharpe2,3,5,
  6. Padraig Warde2,3,5
  1. 1Cancer Care and Epidemiology, Queen's Cancer Research Institute, Kingston General Hospital, Kingston, Ontario, Canada
  2. 2Radiation Treatment Program, Cancer Care Ontario, Toronto, Ontario, Canada
  3. 3Radiation Medicine Program, Princess Margaret Hospital, Toronto, Ontario, Canada
  4. 4Department of Radiation Oncology, Credit Valley Hospital, Mississauga, Ontario, Canada
  5. 5Department of Radiation Oncology, University of Toronto, Toronto, Ontario, Canada
  1. Correspondence to Dr Michael Brundage; michael.brundage{at}krcc.on.ca

Abstract

Objectives To describe current patterns of practice of radiation oncology peer review within a provincial cancer system, identifying barriers and facilitators to its use with the ultimate aim of process improvement.

Design A survey of radiation oncology programmes at provincial cancer centres.

Setting All cancer centres within the province of Ontario, Canada (n=14). These are community-based outpatient facilities overseen by Cancer Care Ontario, the provincial cancer agency.

Participants A delegate from each radiation oncology programme filled out a single survey based on input from their multidisciplinary team.

Outcome measures Rated importance of peer review; current utilisation; format of the peer-review process; organisation and timing; case attributes; outcomes of the peer-review process and perceived barriers and facilitators to expanding peer-review processes.

Results 14 (100%) centres responded. All rated the importance of peer review as at least 8/10 (10=extremely important). Detection of medical error and improvement of planning processes were the highest rated perceived benefits of peer review (each median 9/10). Six centres (43%) reviewed at least 50% of curative cases; four of these centres (29%) conducted peer review in more than 80% of cases treated with curative intent. Fewer than 20% of cases treated with palliative intent were reviewed in most centres. Five centres (36%) reported usually conducting peer review prior to the initiation of treatment. Five centres (36%) recorded the outcomes of peer review on the medical record. Thirteen centres (93%) planned to expand peer-review activities; a critical mass of radiation oncologists was the most important limiting factor (median 6/10).

Conclusions Radiation oncology peer-review practices can vary even within a cancer system with provincial oversight. The application of guidelines and standards for peer-review processes, and monitoring of implementation and outcomes, will require effective knowledge translation activities.

  • Radiotherapy

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/

Statistics from Altmetric.com

Article summary

Article focus

  • Peer review is promoted as a quality-improvement exercise in radiation oncology. Current practice regarding peer review is not known. This research was the first step in a strategic initiative to optimise peer-review activities within a centralised cancer system. The research consisted of a cross-sectional analysis of patterns of peer review in 14 radiotherapy programmes within a provincial cancer system. Existing barriers and potential facilitators to optimal use of peer review in this context were identified.

Key messages

  • Peer review was strongly endorsed in principle.

  • Although some form of peer review currently exists within all surveyed cancer centres, practice variation suggests potential variation in the quality of the peer-review processes.

  • Standards and guidelines for peer-review processes are needed. Recommendations from an ASTRO ‘white paper’ will guide and potentially facilitate development of standards for peer review.

Strengths and limitations of this study

  • Strength of the study was that all cancer centres within a large centralised cancer control system responded to the survey, allowing the full scope of peer review activities within the province of Ontario to be identified along with barriers to its use.

  • The survey findings may not be generalisable to cancer control agencies in other jurisdictions in which unique barriers to use of peer review may exist. The survey recorded self-reported perceptions of the peer-review process; future research should be directed at collecting the direct and indirect outcomes of peer review.

Introduction

High quality of care is important to any medical discipline, but is of particular concern in radiation oncology (RO) treatment programmes given the potential for serious harm in the event of a treatment-related error.1 As modern radiotherapy planning and delivery systems become increasingly sophisticated and complex, quality-assurance procedures must also evolve in order to ensure that consistently effective and safe therapy is delivered.1 ,2 Although much of the attention regarding the safety of radiotherapy has focused on the relationship between quality-assurance practices and critical high-profile dose delivery incidents, quality-assurance programmes are also necessary to minimise the potential for less severe ‘errors’, or inappropriate variations in processes or practice, that represent threats to the overall quality of radiotherapy care.3 While RO has a long history of high-quality technical assurance (eg, processes designed to ensure technical quality, such as valid machine calibration, laser setup calibration and accurate treatment planning systems), the more subjective decisions made by attending physicians are less routinely subject to quality-assurance processing, and are the target of peer review.2 Clearly, both high-quality technical processes and high-quality technical medical care are required to optimise patient outcomes.

Peer review has been defined in a variety of ways in the literature. An often cited broad definition of peer review is ‘the evaluation of creative work or performance by other people in the same field to enhance the quality of work, or the performance’, where the word ‘peer’ refers to people in the same profession who are of the same or higher ranking.4 This definition has been applied to peer review in many disciplines, and in many contexts, including RO.5 Others have suggested the term ‘audit and feedback’.6 These definitions are conceptually broad, and encompass peer-review activities occurring on several levels. For example, tumour board conferences typically review medical decision-making in a specific patient's case, and ensure that the decision to employ radiotherapy (alone or with other modalities) is appropriate. At an organisational level, accreditation processes have been used to ensure adequate structures and processes are in place in an RO programme to support a high level of care and safety.

In this report, we focus on a specific aspect of peer review, namely, the review of individual radiation plans. In this context, a radiation oncologist (alone or as part of a multidisciplinary team) reviews the subjective decisions made by the prescribing oncologist when implementing technical aspects of care, including volume segmentation or contouring, choice of prescribed dose and fractionation, selection of regional anatomy covered and dosimetry optimisation. These activities are distinct from quality-assurance practices in RO that involve other disciplines (eg, plan review by medical physics5 ,7 or by radiation therapist5).

There is evidence that oncologist-to-oncologist peer review is critical to the quality of radiation treatment plans. Studies of observer variation in contouring practices make clear that radiation oncologists can substantially differ in their contouring patterns when presented with an identical case. This variation has been demonstrated in the treatment settings of prostate cancer,8 ,9 lung cancer10 and breast cancer,11 and has prompted the development of contouring guidelines.12 ,13 In the conduct of randomised clinical trials, ‘real-time review’ of treatment plans has consistently detected planning protocol deviations, underscoring how interoncologist planning variation occurs in practice despite the availability of a prescriptive clinical trial protocol. Importantly, this observed variation can have significant negative impact on treatment quality,9 ,14 and on clinical outcomes if not mediated by the peer review process.15 Finally, direct evidence from audit and feedback processes has shown that peer review does detect errors that can be corrected prior to the initiation of treatment. In a large Ontario study of over 1000 patients, errors were detected in approximately 7% of cases, and multivariate analyses showed that the error rate was independent of time of year, experience of the oncologist, or of a postresidency fellow being involved in the case.3 In Australia, regular use of audit and feedback has revealed similar findings, with 6% of plans overall ‘failing’ the audit process.2 Less well described is the extent to which the identified errors were acted on.

Ontario's provincial cancer agency Cancer Care Ontario has identified the need to improve and expand peer-review activities across the Ontario cancer system as a strategic direction of the Provincial Radiation Treatment Program to improve the quality of care. As is the case in most Canadian provinces, RO services are highly centralised in Ontario, where 14 centres serving a population of about 13 million. Cancer centres range in size from relatively small centres (700–1000 new case referrals per year) to two large centres in a large metropolitan area (over 6000 new case referrals per year). Most either participate in, or operate, postgraduate education programmes in RO. In the interests of developing a programme to optimise the use of peer review in the province, we undertook this research to describe current patterns of peer review in RO practices across Ontario and to identify existing barriers and potential facilitators to its expanded and optimal use.

Methods

We undertook a survey of 14 cancer centres using an electronic survey. Survey items were developed by consensus of expert opinion of the project leaders based on a literature review and current knowledge. The survey was pilot tested in one regional cancer centre with an established peer-review programme. The survey, which was sent electronically to the attention of the Heads of RO programmes in Ontario, began with a brief set of instructions, and information regarding the scope of peer review addressed in the survey. A delegate from each cancer centre RO programme filled out a single survey based on input from their multidisciplinary team, which included RO, medical physics and radiation therapy at that centre. Each Head of RO took responsibility for the survey responses, and used delegates at their discretion based on local circumstances. Confidentiality of the responses was ensured. Some survey items asked questions with response options constructed as a Likert scale ranging from 0 to 10, with anchors defined as appropriate for each section of questions (eg, ‘not important’, 0 and ‘extremely important’, 10). Some questions utilised binary (eg, yes/no) responses or categorical responses (eg, ‘almost always’, ‘often’, ‘only sometimes’, ‘rarely or never’) defined as appropriate for the question stem. The survey was approved by the Queen's University research ethics board.

Following pilot testing and subsequent editing to address clarity and redundancy within the questionnaire, the survey was administered by email in April 2011, with responses closing 6 weeks later in May 2011. An introductory letter was included and telephone reminders were used as required.

The survey results were compiled using descriptive statistics, including medians, quartile distributions and ranges where appropriate. Owing to the nature of the study, no inferential statistics were calculated given that no specific hypothesis testing was undertaken.

Results

All 14 (100%) cancer centres responded. All centres performed peer-review activities, although the extent, nature and use of the review varied.

Importance and purpose of peer review

Regarding the importance of peer review, eight (57%) centres ranked the importance as 10/10 (extremely important), with the remaining centres rating the importance as either 8/10 (n=2 centres) or 9/10 (n=4 centres). No centre rated the importance of peer review below eight points.

Figure 1 illustrates respondents’ opinions regarding the importance of each of eight potential reasons for undertaking peer review in RO. Detection of medical error, and improvement of the treatment planning processes or policies, were both rated highly in terms of their importance (median scores of 9 on a scale ranging from 0, ‘not important’ to 10, ‘most important’). IQRs for all of the reasons listed in figure 1 were 5/10 or higher.

Figure 1

Survey results regarding the importance of peer review. The range (horizontal bar), IQR (box) and median score (vertical line) are illustrated.

Current utilisation of peer review at individual centres

Figure 2 illustrates the percentage of plans that were typically peer reviewed across centres. Four centres (29%) reported routine peer review in more than 80% of cases treated with curative intent, and six centres (43%) reported using peer-review processes in at least 50% of curative cases. For curative plans employing IMRT, eight centres (57%) peer reviewed more than 50% of cases. Eleven centres reported that the majority of the curative cases were guided by a local radiation-specific treatment policy. As illustrated in figure 2, peer review in cases treated with palliative intent was performed less frequently.

Figure 2

Stacked bars represent the proportion of centres that report how comprehensive peer-review activities are in curative and palliative cases, respectively.

Format of peer-review practices

Centres reported a number of different formats for conducting peer review. The most common format, used in nine centres (64%) routinely, was the use of peer review in multidisciplinary groups that did not have a site-specific focus. Six centres reported using peer review by site-specific multidisciplinary groups (eg, head and neck cancer cases), and four centres reported the routine use of peer review by an individual (eg, a radiation oncologist reviewing the contouring on a colleague's case).

When peer review was conducted in multidisciplinary groups, all centres reported that radiation oncologists were routinely present, with radiation therapists (n=10, 71%) and medical physicists (n=9, 64%) attending ‘always’ or ‘almost always’. Residents, fellows and other students attended in fewer centres (n=3), and allied health professionals and nurses were rare participants.

Most centres (12 of 14) performed peer review in breast, lung and genitourinary sites, although each centre reported at least one common cancer site that was not yet active in a peer-review programme routinely.

Organisation and timing of peer-review activities

Most centres reported that the organisation of the peer-review processes was carried out by radiation therapists, either exclusively (46%) or in combination with other staff. The process for case selection varied considerably between centres; radiation oncologists identified cases in four centres, medical physicists in three and radiation therapists in six, whereas four centres used a routine mechanism by which cases were identified systematically.

With regard to the timing of peer review, five centres reported ‘always’ or ‘almost always’ conducting peer review prior to the initiation of treatment and two centres reported ‘always’ or ‘almost always’ conducting peer review after starting treatment. The remaining seven centres reported a mixed timing of case review.

Case attributes subjected to peer review

The survey used a case example of a patient receiving radiotherapy for stage IIIB Non-Small Cell Lung Cancer with curative intent. In this context, centres were asked to identify which aspects of the radiation treatment plan were peer reviewed, and 11 centres with established review for lung cancer patients provided data. The results are shown on figure 3. As can be seen, the most common elements of peer review performed were choice of treatment volume, review of DVH’s and contouring of volumes. Many attributes such as pathology review and staging review, the decision to treat (and use of evidence-based guidelines) were undertaken as part of other quality assurance exercises (eg, tumour-board case conferences).

Figure 3

Elements of a curative case (stage III non-small cell lung cancer) that are typically reviewed in each centre. The length of each bar represents the number of centres typically reviewing each element of the plan. CBCT, cone-beam CT; CTV, clinical target volume; DRR, digitally reconstructed radiograph; DVH, dose-volume histogram; GTV, gross tumour volume; PTV, planning target volume.

Outcomes of peer-review activities and their recording

Eleven centres (79%) recorded the outcome of at least one aspect of the peer-review process. Of these, one centre recorded 15–20% of plans requiring some change as a result of peer-review processes, five centres (45%) recorded 5–9% of reviewed plans requiring change, one centre recorded 2–4% of reviewed plans requiring change and three (27%) centres recorded <2% of plans requiring change. Figure 4 illustrates centres’ responses regarding how these outcomes of the peer-review processes were documented. Eleven centres routinely made recommendations from the peer review to the attending oncologist, but fewer recorded these recommendations either off-line or as part of the medical record. Only one centre recorded the response of the attending physician to these recommendations on the medical record. Six centres reported having some concerns about recording the outcomes of peer-review activities as part of the legal medical record.

Figure 4

Bars represent the number of centres that report how each outcome of peer-review processes is documented.

Expanding peer-review activities: perceived barriers and facilitating factors

Thirteen centres (93%) reported plans to expand or enhance peer-review activities. Nine reported a plan to increase the proportion of curative cases reviewed, whereas only two centres had plans to expand activities in cases managed with palliative intent. Eleven centres planned on improving their current processes for conducting peer review.

A number of potential barriers to expanding peer review were rated by each centre on a scale from 0 (not limiting) to 10 (important limiting factor). The results are shown on figure 5. Considerable variation was seen between centres; each item ranged widely in responses, with no single item being rated higher than 6/10 for more than half of the centres. Engaging radiation oncologists in attending peer-review sessions more regularly was the highest rated item.

Figure 5

Survey results regarding barriers to expanding peer-review activities. The range (horizontal bar), IQR (box) and median score (vertical line) are illustrated. MRTT, medical radiation technologist (therapy); PC, personal computing infrastructure; RO, radiation oncologist.

Discussion

Our results illustrate that across a comprehensive provincial RO programme, all centres perform peer review. These activities, which include oncologist-to-oncologist peer review as a minimum criterion, complement other departmental review activities (such as accreditation or tumour boards) by focusing explicitly on review of RO planning. Although some form of peer review occurs at all radiotherapy centres in Ontario, the current patterns of peer-review activities, the intended expansions of peer review and the perceived barriers and facilitators of peer-review activities vary substantially, and in no centre was peer review performed on all cases. This variation implies some potential differences across centres in quality of the peer-review processes, and the impact of those processes on quality of technical care.

All centres strongly endorse peer review in principle. This endorsement was reflected in the universally high-importance ratings of peer-review activities, and in the stated plans for expansion of peer-review activities in most centres. This endorsement is congruent with the recognised benefits of peer review as reviewed earlier.3 ,16 ,17 Our survey results strongly reflect the perceived benefits of peer review for promoting process feedback and improvement, for reinforcing (or identifying needed change in) existing intervention-specific policies, and for detecting medical errors prior to treatment completion. These findings were common across centres.

In contrast, the observed variation in many survey response categories indicates the need for standards and/or guidelines for peer-review processes. Guidelines for case selection, for the case elements/attributes that are highest priority for peer review, and for acceptable timing of peer-review activities would assist centres to improve consistency of practice, prioritisation and optimisation of their efforts. With regard to the timing of peer review, volume segmentation and contouring decisions need to be reviewed prior to treatment dosimetric planning, if replanning of cases (due to identification of required modification) is to be avoided. Peer review of dosimetric judgements (eg, dose-volume histogram results), however, can only be undertaken after treatment planning. Programmes must therefore consider a two-stage peer-review process, or accept that some replanning will be required if a single-stage peer review is utilised following treatment planning decisions. Obviously, peer review should be performed prior to treatment initiation since changes are less likely to be made once treatment starts. Recommendations from an ASTRO white paper18 (one in a series of papers addressing quality-assurance methods for radiotherapy treatment) should be helpful in the development of such standards. Such recommendations, however, will require accompanying strategies for implementation processes and reporting mechanisms in order to be most effective.

A second key area of observed variation between centres relates to documentation of peer-review processes, and action taken based on the results. We did not undertake in the survey to systematically collect outcomes of the peer-review process (which was beyond the scope of many participating centres), but rather, collected survey information on patterns of current data recording. Concerns have been expressed regarding the medical-legal implications of recommendations by peers for changes to prescribed treatment.19 How large a barrier these potential medical–legal issues pose to optimal use of peer-review processes remains unknown in Ontario, and may vary significantly in other jurisdictions depending on their related jurisprudence. Nevertheless, the potential for improvement in the quality of care of individual patients, and more broadly, in the quality of treatment programmes, suggests that clear and permanent documentation of the peer-review components of quality assurance is needed.20 In Ontario, rather than allowing individual documentation practices to evolve with time as more centres gain greater experience with peer-review activities, action will be undertaken at a provincial level to assist centres with policies for appropriate and adequate documentation.

Third, there were other intercentre differences that provoke potential areas of further research. For example, what is the optimal target or benchmark for the proportion of cases to be peer reviewed; a target of 100% ensures the maximal detection of medical error but may not be feasible in some centres, and other objectives (such as reduced variation in care and evaluation of treatment policies) can be achieved by reviewing a subset of cases. For palliative cases specifically, given their often unique characteristics, what are the benefits of increasing peer-review activities in this patient group? Given the teaching opportunities inherent in peer review, do opportunities for research in medical education exist? What are the incremental costs for implementing peer-review activities? What effect does a local champion have in establishing and maintaining the importance of peer review within the department? What are the advantages of having multiple disciplines involved in the review?

Finally, our survey identified facilitators and barriers to peer review; this provides an opportunity to act on research findings, reinforcing facilitators and addressing barriers. Knowledge translation activities will be initiated by the provincial programme including: acknowledging and remediating the barriers described by survey respondents; clarifying the relationships between peer-review exercises (eg, the decision to treat review by a multidisciplinary tumour board); establishing quality indicators and measures to monitor system performance; assessing progress in peer-review initiatives routinely at provincial meetings of clinic programme heads; and exploring the use of technology to improve consistency21 and facilitate processes (eg, use of explicit processes such as an Audit tool16). With regard to the barrier of costs to treatment programmes, to date in Ontario, these costs have primarily been absorbed by existing operational budgets. Further research will determine the direct and indirect costs of peer-review activities; defining these costs might allow for incremental funding for these activities.

Conclusion

Review of radiation treatment planning decisions by radiation oncologists within their clinical peer group is being undertaken in all radiation treatment centres within a large Canadian jurisdiction (Ontario). The strong support for peer review and widespread plans to expand peer-review activities is encouraging, but standards for best practices are required. Plans are currently underway to develop guidelines and standards for peer-review activities and policies in Ontario, aligned with those of other professional organisations such as ASTRO. Many of the identified facilitators and barriers will be common to other jurisdictions or organisations providing RO services, and as such, can provide those agencies with a basis for implementing peer-review quality-improvement activities (although additional unique barriers may exist in a particular clinic context). Research linking peer-review practices with improved patient outcomes is recommended.

Acknowledgments

We acknowledge Ms Elizabeth Lockhart and Ms Brenda Bass for their invaluable administrative assistance in conducting this project. We also acknowledge each of the radiation oncology programme leads at Cancer Care Ontario Centers for providing information specific to their programme and for feedback regarding the interpretation of the survey results: Dr C de Metz, Dr M Gospodarowicz, Dr F Iqbal, Dr J Kamra, Dr M Lock, R Rachakonda, Dr K Schneider, Dr S Shehata and Dr S Gulavita. W Wells, S Wong, J Wright and Dr Brundage are supported by a research chair award from Cancer Care Ontario.

References

View Abstract

Footnotes

  • Contributors All authors participated in the planning of the study, the design of the questionnaire, the analysis and interpretation of the findings. All authors read and approved the final manuscript. MB takes responsibility for the accuracy of the report.

  • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests MB was the PI of the study and the lead writer. SF was colead of the overall project and the radiation therapist lead provincially. TM was a radiation oncologist coinvestigator, and head of radiation oncology at a participating centre. EG was the provincial programme administrator, and PW the provincial programme leader.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.