Article Text

Original research
Exploring the value of structured narrative feedback within the Serious Illness Conversation-Evaluation Exercise (SIC-Ex): a qualitative analysis
  1. Jenny Ko1,
  2. Amanda Roze des Ordons2,
  3. Mark Ballard3,
  4. Tamara Shenkier4,
  5. Jessica E Simon5,
  6. Gillian Fyles6,
  7. Shilo Lefresne7,
  8. Philippa Hawley8,
  9. Charlie Chen5,
  10. Michael McKenzie7,
  11. Justin Sanders9,
  12. Rachelle Bernacki10
  1. 1Department of Medical Oncology, BC Cancer Agency Abbostford Centre, Abbotsford, British Columbia, Canada
  2. 2Department of Critical Care Medicine and Division of Palliative Medicine; Department of Anesthesiology, University of Calgary Cumming School of Medicine, Calgary, Alberta, Canada
  3. 3Department of Internal Medicine, Chilliwack General Hospital, Chilliwack, British Columbia, Canada
  4. 4Department of Medical Oncology, BC Cancer Agency Vancouver Centre, Vancouver, British Columbia, Canada
  5. 5Department of Oncology and Community Health Sciences, University of Calgary Cumming School of Medicine, Calgary, Alberta, Canada
  6. 6Pain and Symptom Management/Palliative Care Program, BC Cancer Agency Sindi Ahluwalia Hawkins Centre for the Southern Interior, Kelowna, British Columbia, Canada
  7. 7Department of Radiation Oncology, BC Cancer Agency Vancouver Centre, Vancouver, British Columbia, Canada
  8. 8Department of Palliative Care, BC Cancer Agency, Vancouver, British Columbia, Canada
  9. 9Department of Palliative Care, McGill University, Montreal, Quebec, Canada
  10. 10Department of Palliative Care, Dana-Farber Cancer Institute, Boston, Massachusetts, USA
  1. Correspondence to Dr Jenny Ko; Jenny.Ko{at}bccancer.bc.ca

Abstract

Objectives The Serious Illness Conversation Guide (SICG) has emerged as a framework for conversations with patients with a serious illness diagnosis. This study reports on narratives generated from open-ended questions of a novel assessment tool, the Serious Illness Conversation-Evaluation Exercise (SIC-Ex), to assess resident-led conversations with patients in oncology outpatient clinics.

Design Qualitative study using template analysis.

Setting Three academic cancer centres in Canada.

Participants 7 resident physicians (trainees), 7 patients from outpatient cancer clinics, 10 preceptors (raters) consisting of medical oncologists, palliative care physicians and radiation oncologists.

Interventions Each trainee conducted an SIC with a patient, which was videotaped. The raters watched the videos and evaluated each trainee using the novel SIC-Ex and the reference Calgary-Cambridge Guide (CCG) initially and again 3 months later. Two independent coders used template analysis to code the raters’ narrative comments and identify themes/subthemes.

Outcome measures How narrative comments aligned with elements of the CCG and SICG.

Results Template analysis yielded four themes: adhering to SICG, engaging patients and family members, conversation management and being mindful of demeanour. Narrative comments identified numerous verbal and non-verbal elements essential to SICG. Some comments addressing general skills in engaging patients/families and managing the conversation (eg, setting agenda, introduction, planning, exploring, non-verbal communication) related to both the CCG and SICG, whereas other comments such as identifying substitute decision maker(s), affirming commitment and introducing Advance Care Planning were specific to the SICG.

Conclusions Narrative comments generated by SIC-Ex provided detailed and nuanced insights into trainees’ competence in SIC, beyond the numerical ratings of SIC-Ex and the general communication skills outlined in the CCG, and may contribute to a more fulsome assessment of SIC skills.

  • PALLIATIVE CARE
  • MEDICAL EDUCATION & TRAINING
  • ONCOLOGY

Data availability statement

Data are available on reasonable request. Data underlying this study may be made available to other researchers on request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • This study examined how resident physicians’ performance in Serious Illness Conversations (SICs) with outpatient cancer patients is evaluated using the novel SIC-Evaluation Exercise (SIC-Ex) assessment tool.

  • Narrative comments in the SIC-Ex allowed assessment of the nuanced application of skills that may not be readily captured in numerical scales.

  • Limitations include a small sample size and limited preintervention training in the SIC framework for trainees and raters.

Introduction

The Serious Illness Conversation (SIC) Programme, developed by Ariadne Labs, is a system-level intervention to improve the prevalence, timing and quality of advance care planning (ACP) in serious illness.1 2 It guides implementation of communication skills training and systems changes that support adoption and use of clinical conversation tools.1 2 The Serious Illness Conversation Guide (SICG), the signature clinical tool of the programme, helps clinicians share prognosis and elicit critical information to inform future decision-making and care. Its scope includes illness understanding, decision-making preferences, goals and fears, views on trade-offs and impaired function, and caregiver involvement.3–6 A number of health systems have adopted the SICG as SICG as the basic framework for ACP discussions in cancer clinics, including those in which this study was situated.

Prior studies show that the quality of documented conversations regarding serious illness improve significantly and measurably after implementation of the SICG framework.4 5 7 8 Despite the known benefits of SIC, clinicians in a busy medical practice may encounter challenges in routinely having these conversations as well as teaching them to trainees. Common assessment tools to evaluate a trainee’s competence in communication skills may not necessarily capture their competence in SIC. To our knowledge, no assessment tool has yet been developed or tested specifically for assessment of trainee competency in SIC with patients who have a cancer diagnosis. The value of free-text narrative comments in developing trainees’ SIC skills is unknown, and there are concerns that a simple list of communication skills on a generic medical interview assessment form may not capture the nuances and interactions that SIC may generate.

We developed a new assessment tool to evaluate trainees’ skills in conducting SIC. In our previous publication, we have described in detail the steps in developing the SIC-Evaluation Exercise (SIC-Ex) based on the SICG framework developed by Ariadne Labs,9 10 including the use of Kane’s framework for generating validity evidence.10

Objectives

The objective of this report was to explore how narrative comments within the SIC-Ex may contribute to the assessment of resident-led SIC.

Methods

We conducted the study between 1 September 2016 and 16 March 2017. We recruited 7 resident trainees, 7 patients and 10 preceptors (rater below) from 3 cancer centres in Canada. Trainees were recruited from family practice and medical oncology residency programmes. Patients were recruited from outpatient medical oncology clinics at two cancer centres where the raters worked. We aimed to recruit 10 trainees, 10 patients and 10 raters, however, only 7 trainees and 7 patients volunteered to participate.

Trainees and patients who volunteered for the study provided informed consent. Patients were briefed about the purpose of the study and that they would be videotaped without their face or other identifying information being recorded. Each trainee was briefed about the purpose of the study and that they would be videotaped to show their facial expressions and body language as well as the recording of their conversations. Trainees received a 15 min teleconferenced orientation about SICG as well as a sample of SIC-Ex, SICG reference materials and framework (revised version available at https://www.ariadnelabs.org/tools-and-downloads/), and a 10 min in-person orientation on the day of the study. Raters had two teleconferenced hour-long orientation sessions to introduce them to the study and SIC, but did not receive specific training in SICG.

On the day of the study, each trainee came to an outpatient medical oncology clinic where they met with a designated patient. The trainees then conducted a videotaped Serious Illness Conversation (SIC) with the patient. Each rater watched all the videos and evaluated each trainee using SIC-Ex (experimental tool) as well as Calgary-Cambridge Guide11 (CCG; reference tool) (online supplemental appendix). The CCG was chosen as a reference tool as it is a validated, widely used and well-known evaluation tool to evaluate trainee communication skills, with several domains comparable to those in the SIC-Ex—Initiating the session; Gathering information; Providing structure; Building relationship; Explanation and planning; Closing the session; Options in explanation and planning. The same raters watched the videos again 3 months later and rated them using the same tools.

This report focuses on the written narrative comments generated from the raters while completing the SIC-Ex in response to the following open-ended questions: ‘what was done well?’, ‘what could be improved?’ and ‘other comments’.

We performed a thematic analysis of the narrative comments using the approach of template analysis. To analyse the raters’ narrative comments for each trainee, two independent coders, who were also raters, used template analysis to code the raters’ narrative comments. We first developed a coding template with a priori themes identified by the two coders. The themes were organised into the categories of general skills, such as ‘adhering to the SICG’, and more specific skills such as ‘identifying substitute decision maker(s)’ or ‘affirming commitment’ (table 1). The analysis was performed manually without use of a software program.

Table 1

Themes from the rater feedback to trainees using the SICG

To enhance the credibility of analysis, each of the two coders independently reviewed and coded the narrative comments and codeveloped a preliminary coding template based on multiple readings. The coders indexed and mapped (ie, categorised) the codes into themes and subthemes through multiple iterations of discussions and through which disagreements were resolved. All names and identifying information were encrypted, and all experts were anonymised. All data were stored and transmitted through a secure server.

A time gap between the initial conduct of study and publication of the manuscript is noted. Due to COVID-19 pandemic and limited clinical resources, the authors only recently had capacity to prepare the current manuscript.

Patient and public involvement

None.

Results

Seven trainees including five postgraduate year-1 residents from family medicine and two residents from medical oncology participated. There were seven patients—two with lung cancer, one with lymphoma, one with renal cell carcinoma, two with breast cancer, one with malignant mesothelioma. The 10 raters consisted of 6 palliative care physicians, 2 medical oncologists and 2 radiation oncologists.

Narrative comments provided by raters explored numerous verbal and non-verbal attributes of resident behaviours during the conversation. Some comments mapped to both SICG and CCG (eg, empathy, open-ended questions, clarity, exploration, planning), whereas others were specific to a given framework.

Template analysis yielded four themes, each with a number of subthemes (table 1).

Adhering to the SICG

This theme illustrated behaviours that followed the structure and language of the SICG. Subthemes identified were (a) identifying substitute decision maker(s), (b) affirming commitment and (c) introducing ACP. Raters indicated that they used specific components of the SICG to evaluate trainee skills and offer feedback, for example,

”Try using some of the Guide language (rather than paraphrasing) and structure when you are first having these conversations and see if that helps you (or not)…”

Engaging patients and family members

Subthemes included (a) using open-ended questions, (b) avoiding interruptions, (c) minimising explanation, (d) allowing patients to talk and explore and (e) building relationship with patient/family. The raters’ comments within this theme highlighted avoiding interruption, minimising explanations, using open-ended questions, and allowing patients to talk and explore, as key to effective communication.

Raters considered asking open-ended questions as an important component of general information gathering under CCG. Explanation, in contrast, emerged as problematic when the trainee provided explanations rather than further exploring the patient’s perspective. The raters provided feedback to reflect this, such as:

“(You asked) Good questions: eg ‘what would that balance look like?’ … Avoid explanations during patient response; (Please offer) fewer explanations (to the patient).”

The raters also commented on trainees’ ability to provide structure; attributes such as good flow of the conversation and time management skills were highlighted as desirable skills. As an example, one rater recommended that

”transition between open—and close-ended questions can be smoothed; (she/he) tends to ask two (or three) questions at a time, should do one at a time and allow patient time to answer.”

The raters emphasised process skills such as clarifying patients’ statements, clarity in language and active listening. The following attributes related to building the relationship were felt to relate to CCG and represent components of building a competent medical interview: engaging, non-judgemental, respect, rapport, silence, interest in person/curiosity, recognising/responding to cues, normalising/validating, use of humour, support, encouraging, reframing/counselling, and importantly, avoiding interruption.

Building the relationship involved engaging family members as well as patients. Raters emphasised the importance of involving a patient’s family in SIC and encouraged trainees to include this as part of their approach. This encouragement was conveyed through providing feedback that reinforced this desired behaviour, such as ‘Good job engaging patient’s partner.’

Conversation management

Subthemes included in this theme are: (a) initiating the session, (b) gathering information relevant to SIC, (c) process skills and (d) allowing discussion of next steps. The raters noted how trainees initiated the session, including whether an introduction and agenda were included.

“Even though had previously met patient, would be helpful to introduce self and role as patient may have forgotten.”

Specific to information gathering by trainees, the raters recommended techniques such as open-ended questions that would encourage ‘(further) drawing out what patient would like to discuss (in relation to SIC).’

Planning for next steps was a component of SIC and CCG which raters noted was overlooked or omitted by some trainees, and commented that

”discussion of next steps would have been helpful… Ran out of time otherwise probably would have discussed planning elements of ACP.”

Process skills such as exploring, reflecting/paraphrasing, summarising, individualising/contextualising recurred as essential aspects of SIC and also CCG. Non-verbal communication and empathy emerged as part of building the relationship in the context of SIC.

Being mindful of demeanour

Subthemes such as kindness, compassion, respect, approachability, calmness, engaging manner, self-awareness, comfort, confidence, judgement, learner skills described the trainees’ behaviours during the conversations. For example, one rater wrote

“(the trainee) seems a bit uncomfortable asking some difficult questions leading to vague questions that I think was sometimes confusing to patient.”

Another rater also commented, in regards to a trainee’s pre-existing knowledge and skills, that

‘With more oncological knowledge, learner probably could have been more specific about certain medical interventions, but for level of training, did very well.’

Some raters noted that some narrative comments were misclassified, such as describing criticisms under ‘what was done well’ and praises/criticisms under ‘other comments’. Feedback was provided, often under ‘other comments’, on the language of the SIC-Ex questions, on the study setup, and the audio/video setup.

”Video started after initial introduction so couldn’t comment on that, and am unable to comment on documentation or ongoing care.”

Discussion

To our knowledge, this is the first study to assess narrative feedback to trainees conducting SIC with outpatient oncology patients as part of the SIC-Ex. Prior studies showed significant reliability and added value of narrative comments in assessing residents’ skills in performing medical interviews.12–14 Many pre-existing evaluation tools, including CCG, do not inherently include space for narrative comments. While the CCG and SIC-Ex overlapped in assessing some of the same elements of communication, we found that the SIC-Ex allowed for more nuanced evaluation of resident competencies, specifically related to eliciting patient’s wishes and values. The value of narrative comments provides additional validity evidence for adopting the SIC-Ex in teaching and assessing resident competence in discussing serious illness with patients in outpatient oncology clinics.

The SIC-Ex enables structured narrative comments aimed at improving the competencies required for effective SIC. In a previous study, use of silence and limiting clinician speaking time to less than 50% of the conversation have been identified as enhancing patients’ experiences in conversations guided by the SICG.15 In our study, this was captured within the subthemes ‘avoiding explanations/interruptions‘ and ‘non-verbal communications‘ and identified as important ways of optimising these conversation. Components of SIC such as engaging family, identifying substitute decision-maker and introducing ACP are important elements of SIC not included in other evaluation tools. The structured narrative component of the SIC-Ex facilitates more nuanced formative feedback for trainees.

Template analysis revealed where the SIC-Ex and CCG overlapped and where they differed. The SIC-Ex enabled raters to offer feedback on skills specific to SIC as well as general communication skills by providing structured prompts to guide narrative comments. Neither the SICG nor CCG were initially developed as assessment tools; however, they are now widely used to assess competence in communication skills. The CCG was designed to guide medical interviews more generally, whereas the SICG is more specific to SIC.2 4 10 16–20 Many of the verbal and non-verbal microskills highlighted by the CCG21 were captured within the narrative comments of the SIC-Ex, suggesting that including a structured narrative component within the SIC-Ex facilitates feedback on communication skills in general as well as those specific to the SICG. This also suggests that numerical rating component of the SIC-Ex could be expanded to include some of the attributes of the CCG to provide a more comprehensive assessment.

Narrative comments identified behaviours that did not map to either the CCG or SIC-Ex, providing the opportunity for more nuanced feedback on competence in SIC. These comments may relate to trainees’ pre-existing communication styles and may not be captured in checklist-based Objective Structured Clinical Examinations that adopt CCG or mini-CEx frameworks.21 22 This domain may overlap with aspects of professionalism.23 Literature suggests that a strong patient–physician relationship is a critical component in successful SICs.15 Further studies might further explore associations between previous professionalism training and competence in SIC.

SIC-Ex provides a useful assessment tool that facilitates iterative and formative feedback on communication around serious illness within competency-based medical education (CBME) frameworks. The structured narrative component of the SIC-Ex in particular may provide useful information for trainees and their programmes as to their skills related to milestones specific to SIC and communication skills more generally.24 CBME focuses on outcomes and abilities to guide the content and assessment of education. Competence is considered as a global assessment rather than focusing on isolated elements of knowledge or skill. Our study10 highlights the importance of including both a numerical scale and narrative comments in enriching the assessment of SIC competencies and guiding meaningful feedback to trainees.

Limitations of this study include a small number of patients, raters and trainees, as well as the absence of a ‘gold standard’ to which to compare the qualitative component. The narrative comments themselves, however, provided guidance on how the SIC-Ex may be further developed as a tool. Another limitation was that trainees and raters in the study received an orientation of the SICG but not extensive training. With varying degrees of familiarity with SIC, trainees may have had challenges with processing unfamiliar components of SIC while conducting a medical interview. Although all of the raters were familiar with SICG, a more detailed orientation to SICG may have generated more in-depth free-text narrative. The comments on the language of the questions and the audio/video setup further identified limitations of the study. Misclassified rater comments suggest the need to familiarise raters on the use of SIC-Ex and modify the prompts for narrative comments. Our study did not include patient feedback on trainees’ communication skills, which may have been valuable for trainees as well as provided further insights into how the SIC-Ex may be modified. We also did not elicit trainees’ self-assessment of their SIC skills or their reactions and responses to the narrative comments.

We reviewed more than 20 pre-existing evaluation tools in developing the SIC-Ex, and selected CCG as the comparator as it was the most evidence-based and aligned with our research goals. Further studies are needed to examine the value of various SIC tools in specific clinical contexts. Concordance between the conversation and its documentation was not evaluated.14 Future larger studies will add validity evidence for the SIC-Ex and its application within specific contexts.

Conclusion

This study demonstrates the value of the structured narrative component of the SIC-Ex, lending further validity evidence to the use of SIC-Ex for assessment of resident trainees’ competence in serious illness communication. Future studies that include more rigorous trainee and rater training and which elicit patient and trainee perspectives will be important in providing further evidence and in further optimising the SIC-Ex as an assessment tool.

Data availability statement

Data are available on reasonable request. Data underlying this study may be made available to other researchers on request.

Ethics statements

Patient consent for publication

Ethics approval

This study involves human participants and was approved by BC Cancer Research Ethics Board (H16-00164). Participants gave informed consent to participate in the study before taking part.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors JK was involved in conceptualisation, methodology, data collection, analysis, writing, visualisation, supervision, project administration, funding acquisition. JK is the guarantor of the study. ARdO was involved in conceptualisation, methodology, data collection, analysis, review and editing. MB was involved in conceptualisation, methodology, data collection, analysis, review and editing. TS was involved in methodology, data collection, analysis, review and editing. JES was involved in conceptualisation, methodology, data collection, analysis, review and editing. GF was involved in methodology, data collection, analysis, review and editing. SL was involved in methodology, data collection, analysis, review and editing. PH was involved in methodology, data collection, analysis, review and editing. CC was involved in methodology, data collection, analysis, review and editing. MM was involved in methodology, data collection, analysis, review and editing. JS was involved in methodology, analysis, review and editing. RB was involved in methodology, analysis, review and editing.

  • Funding This study was funded by Medical Council of Canada Research in Clinical Assessment Grant (# MCC-8/1617).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.