Qualitative methods are an increasing element of the development of randomised controlled trials (RCTs), particularly those of complex interventions where the conduct of the intervention is mediated by human behaviour. Qualitative methods are advocated in the pre-trial development phase of complex interventions and during and after the trial to facilitate interpretations of trial results.^1,2 These recommendations form part of the guidance for the use of qualitative methods in RCTs in the Medical Research Council guidance, A framework for development and evaluation of RCTs of complex interventions to improve health.³ In this paper, we illustrate an additional use for qualitative methods—that is, qualitative methods used alongside a trial may provide evidence to intervene in a trial if necessary, including evidence for the rationale to discontinue a trial intervention arm. Although qualitative evidence has been used elsewhere to inform the discontinuation/continuation of a drug trial,⁴ this is the first reported use of an integrated and coterminous qualitative process evaluation to inform the conduct and continuation of an RCT of a complex behavioural intervention. We present here qualitative analysis derived from a qualitative study of the Decision Analysis in Routine Treatment Study II (DARTSII) efficacy trial. This analysis allowed us to make a decision to discontinue one arm of the trial on the basis that a major component of the intervention, a standard gamble values elicitation exercise, was confusing to most participants. Further, the outputs of the standard gamble did not validly represent participant values as intended. We argue that without the qualitative evidence we might not have been able to identify the problem at an early stage and, crucially, we would not have been able to justify a decision to suspend that arm.

METHODS

The efficacy trial

To assist both doctors and patients to enact evidence-based shared decision making, a computerised tool, DARTSII, was developed for patients with atrial fibrillation (AF). The development of the DARTSII decision support tool and the conduct of the RCT with its use have been described elsewhere.⁵

A key tenet of shared decision making is the need to incorporate patient values into decision-making models.⁵ Mechanisms to elicit patient values have built upon economic methods that focus on the construction of individual utility. Standard gamble and time trade-off methods for elicitation of values have been used in decision aids in healthcare.^6,7 Although there is limited experience of use of standard gamble methods in a shared decision-making context, problems using the standard gamble in other contexts are reported in the economics literature.⁸ In the DARTSII, the explicit elicitation of participant values was conducted using the standard gamble method to inform the decision analytical phase of the decision support tool (fig 1).⁵ In the standard gamble component of the intervention, participants were presented with a series of hypothetical options for health states—for example, a choice between (A) the certainty of a mild stroke or (B) a gamble, a 50% risk of death, with a corollary 50% chance of normal health (fig 1). The hypothetical options presented to participants then changed with option A remaining the same, and the risk of death changing in comparison to the risk of normal health in option B. The purpose of this exercise was to find the point of indifference between two choices, in this case two health states: option A (eg, certain stroke) and option B (a gamble, full health or death). At the point at which the participant can no longer decide between options A and B, a value between 0 and 1 is derived demonstrating patient values for the health states, where 0 = death and 1 = normal health. This value, described as a utility for the health state, is then built into the decision analysis as a measure of the participant’s values regarding health.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

 Sample entry screen for standard gamble exercise.

The pilot study accompanying the development phase demonstrated good levels of acceptability and satisfaction with the tool, and confidence in its usability.⁵ We note, however, that the role of participants in the development phase was to help develop the tool; an inevitably positive orientation to the intervention in contrast with the participants in the trial, who had a range of reasons for engagement related to testing the tool.⁹ An RCT of the DARTSII tool comprising three arms, called here explicit, implicit and guidelines arms (box 1), was subsequently undertaken¹⁰. Each arm of the trial was delivered by a single doctor. The doctors involved in the trial received training in using the computerised decision support tool or the guidelines in their respective arm of the trial. Individual participants were randomly allocated to one of the arms.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

 Graphical representation of the components of, and the relationship between, the Decision Analysis in Routine Treatment Study II (DARTSII) trial and the Thematic Observational Analysis of DARTSII (TOAD), the qualitative process evaluation.

Qualitative study of the RCT

A parallel qualitative study, Thematic Observational Analysis of DARTSII, was conducted alongside the RCT of the DARTSII decision support tool¹⁰ (fig 2). Multiple methods were used to understand the interactional processes of the trial consultations and participants’ experiences and understandings of the trial and of any advice they were given. The first 30 participants recruited to the RCT were invited to take part in the qualitative study: explicit arm, n = 8; implicit arm, n = 11; guidelines arm, n = 11. With participants’ consent, consultations (n = 29) across the three arms of the trial were video recorded. The video camera was positioned statically without an operator so that the upper bodies of the doctor and the participant, and the DARTSII tool (in arms one and two), were visible.

Participants subsequently took part in two semistructured interviews. Within 5 days of the consultation, participants (n = 30) were interviewed about general issues related to their experience of AF, their experience of the consultation and their understanding of how, and what, treatment decisions were reached within it. Participants (n = 26) were interviewed for a second time 90–100 days after the consultation. This interview elicited participants’ views of the specific consequences they attributed to the consultation, their post hoc evaluation of the decision reached and the extent to which they believed that their expectations had been met. Twenty-eight of the first and all of the second semistructured interviews were audio taped and transcribed with the respondent’s consent.

Transcripts of the consultations and interviews formed the formal data for analysis, with field notes being used in the two cases where permission to audio tape the interview was not given. A constant comparative approach to the qualitative analysis of text-based data was used,¹¹ facilitated by QSR Nudist Software. Video recordings were analysed using observational methods.¹² Data were interpreted iteratively by a team experienced in qualitative methods (RHG, BRH, CRM, MJM and TR). Thematic categories within interview and video transcripts were identified by category mapping and comparison.¹¹ After identification of a problem in the explicit arm of the trial, comparative analysis of themes emerging from interview and video transcripts with observation of the videos was conducted by BRH, MJM and TR. Analysis was discussed and challenged in two joint meetings of the qualitative and RCT teams.

The RCT and qualitative studies were carried out by two teams (RCT comprising MPE, MJM, LS, RGT (PI); TOAD comprising RHG, BRH, EFK, CRM (PI), MJM, TR) that were funded and managed separately and worked in collaboration. Ethical approval was granted for each study by the local research ethics committee.

RESULTS

Concerns about the participants’ use of the standard gamble exercise were first raised with researchers by a clinic doctor. He was worried that participants did not grasp the purpose of the exercise and reported difficulties working through the standard gamble with them. Initial qualitative analysis of the consultation videos and concomitant analysis of the interviews revealed that participants were confused about the use and purpose of the standard gamble. Further in-depth analysis of video and interview data (BRH, MJM, TR) confirmed that participants experienced problems with both understanding and carrying out the standard gamble. We present this analysis below. We examined the videos and post hoc interviews for confirming and disconfirming examples of these problems and determined that six of eight participants in this explicit arm experienced these problems, and were unable to carry out the standard gamble exercise. Two participants who did “get it” did so immediately and were able to undertake nuanced discussions of risk and risk calculations. Restrictions of space do not allow us to present extracts for each participant. Rather, representative examples are used below to explain our analysis of the problems encountered by participants. We have used the following transcription system in this paper: Words in double brackets—for example, “((laughs))”—contain author’s comments and generally refer to non-verbal actions of the participants. Pauses are marked in tenths of a second—for example, “(0.5)” is a 0.5 s silence and “(1.0)” is a 1 s silence. With very short turns, acknowledgement tokens like “Yeah” and “Right”, square brackets are used to denote these brief enunciations—for example, “[Yeah]”—and are positioned within the text of the other speakers’ ongoing turn of talk. Pseudonyms are used throughout.

In the extract below, the clinic doctor describes to one of the participants, Geoff, the options available within the standard gamble component. At the beginning of the consultation, the clinic doctor gave a full explanation of the standard gamble component, reiterating this during the consultation. The hypothetical options presented here are between (A) the certainty of a gastrointestinal bleed and (B) shifting odds of normal health or mild stroke. In reiterating the description of the options, the clinic doctor finished by checking Geoff’s understanding. That the options are unclear and somewhat confusing to Geoff is evident both in his response to the question and in his making absolute the odds in option B in his question of clarification, “do I have to take the mild stroke and, to have normal health?” (table 1).

Later in the consultation, Geoff stated his confusion about the content of the standard gamble. He expressed difficulty comprehending the new set of options placed before him (table 2).

In interview, Geoff described his difficulty in understanding the standard gamble as part of a generalised difficulty with computers: “Well I don’t know I understand them to be honest. I cannot even put the bloody thing on and that. I’m hopeless at that.” We cannot conclude, however, from this that computers are the cause of Geoff’s difficulty in understanding the standard gamble; this statement is arguably a face-saving strategy,¹³ since not understanding computers (a technical inability) may be a more acceptable explanation than not understanding the content and the process (a cognitive inability).

Another account of not understanding the standard gamble exercise comes from Anne (table 3).

Anne stated that she found the exercise “baffling”. Of importance in this extract is Anne’s account of “just guessing”. The extract gives the sense in which Anne is simply trying to get to the end of the process and perhaps saying what will get her to that end more quickly. In the video observation Anne is visibly agitated, and in the latter part of the standard gamble she was increasingly ill at ease, seeking reassurance from the carer she brought with her and at moments clasping her head in her hands. Like Geoff, in the interview, Anne looks for explanations for her inability to understand the standard gamble: being “too old” and her “memory wasn’t working”.

Problems with understanding and carrying out the standard gamble clearly caused confusion to some participants, and in two participants, Anne and Paul (below), this was manifest visibly in the videos of consultations. One possibility is that participants find it difficult to separate the concept of the hypothetical risks from their own individual risk. In considering the options during the standard gamble where she verbalised her choices, Anne talked as though the options related directly to her rather than being a hypothetical situation, and was reminded on a number of occasions by the clinic doctor and her carer that these were hypothetical examples and not about her personal clinical circumstances. A further demonstration of this conflation of the hypothetical and the real is evident in the extract from Paul’s consultation below. In the discussion of mild or severe stroke, Paul refers back to a personal experience of stroke with a family member. This description becomes the lynchpin for his subsequent responses to the options available (table 4).

Paul’s firm conviction that he would rather die than have a stroke, be it mild or severe, evidently expresses his strongly held values. At times during the interchange, Paul appeared to be sweating profusely, mopping his brow with a white handkerchief at frequent intervals. His agitation was further evidenced in his manner of increasingly rocking backwards and forwards in his seat towards and away from the computer as the standard gamble exercise progressed. He did not exhibit these behaviours either in the opening or in the later phases of the clinic consultation; rather, he sat back in his chair, nodding affirmations to the clinic doctor. Paul took approximately 20 min to complete the standard gamble component of the tool, in contrast to Dave (below) who completed the standard gamble in approximately 11 min.

Dave and another participant seemed to have little difficulty with the standard gamble exercise. Dave explained his aptitude for the exercise through his understanding of playing the odds as a gambler (table 5).

For Dave the exercise made sense and, it could be argued, produced a measure of his values that was derived from their deliberate assessment of his value states (table 6).

Here, Dave orients to the standard gamble exercise as an exercise in “odds”, which provide “an option”. This contrasts with Geoff, Anne and Paul who orient to the standard gamble as personalised to them. Although Dave was able to carry out the standard gamble, it is not necessary that he understood its purpose or relationship with the decisions he was making. Indeed, he attributed it to the computer in making these decisions (table 7).

On the basis of the analysis of the videos and interviews, there was considerable discussion about the importance and implications of these findings among members of the qualitative study team. Informal detailed discussion between members of the qualitative team and the principal investigator for the RCT led to formal discussion of the issue, incorporating the qualitative evidence, at one of the regular joint meetings of the qualitative and RCT research teams. At this meeting, the decision was taken to discontinue the explicit arm of the trial on the basis of the qualitative analysis, which demonstrated that the standard gamble value-elicitation exercise was causing confusion and was unlikely to produce valid data on patient values. It was believed that it would be unethical to continue, and also that the results would be distorted or impossible to interpret. Approval for revisions to the protocol was granted by the funding body and the local ethics committee.

DISCUSSION

The process of identifying a problem in the explicit arm of the trial was an iterative one, occurring across multiple analyses of video-based data, consultation and post hoc interview transcripts and conversations by, within and between members of the qualitative study team and the RCT team. The initial identification of the problem using the explicit arm of the trial was raised by one of the clinic doctors in the RCT, but the opinions of one individual were not themselves sufficient evidence to warrant cessation of that arm of the trial. We required evidence from the trial itself. Careful analysis of participants’ use of the decision support tool allowed us to demonstrate the specific problems encountered by these participants, and to examine their potential effects on validity in the trial. Regarding the decision to suspend the explicit arm, the two teams worked in concert to address complex methodological and ethical questions raised by participants’ reactions to the values elicitation exercise, the standard gamble.

It was clear that most (6/8) participants were unable to easily carry out the values elicitation exercise. While two participants could carry out the standard gamble exercise, none appeared to understand the purpose of the elicitation of values in relation to the decision-making tool. Understanding is of course produced in the communication and interaction between the doctor and the participant. Nonetheless, as we illustrate in the extracts above, the explanations of the standard gamble, although accurate, are not easy to understand. On the basis of our analysis, we cannot be sure that the numerical results produced by this instrument would accurately reflect participants’ values. This raised questions about how validly to interpret these results in the trial analysis. Furthermore, all participants who were unable to easily carry out the standard gamble exercise exhibited signs of discomfort and/or agitation; two participants palpably manifested anxieties during the consultation. For the research teams this raised ethical issues about duty of care for the participants.

Methodological issues

The decision to cease the explicit arm of the trial was made for methodological and ethical reasons. The decision was made at an early stage, after there were only eight participants in the discontinued arm. These may not have been representative of all potential participants. Nonetheless, we believed that the findings threw sufficient doubt on the application of the intervention that there was an imperative to act rather than submit others to an intervention that was at best confusing. Any results would have been produced by an apparently malfunctioning intervention and we believed that continuation of this would be unethical.

One explanation for the findings might be that the arms of the trial were not carried out effectively or that the interventions were flawed to begin with. However, an extensive development process was undertaken and, as described above, the tools were shown to be acceptable and have utility.⁵ Furthermore, trial doctors were given thorough training in the conduct of each arm of the trial. It is possible that the participants involved in the development of the trial were different from those in the trial itself: participants in the development phases were selected by their general practitioners, whereas those in the RCT were selected from the population of patients with AF. Moreover, participants in each phase consented to different types of involvement, which may have introduced selection bias on the basis of participant motivation for involvement. In the development phase, the task for participants was different from those in the trial. Hence, the consent processes and the expectations of participants were different from those at the development stage. This may explain why the problems identified in the trial were not apparent in the development and pilot phases. Further, as we have demonstrated elsewhere, the context of the research clinic setting and the RCT itself produces different orientations to the study and to the decision support tool, and this influences the conduct of the trial in ways that cannot be predicted or reproduced in a pilot study.^9,14

This study does not enable us to conclude that the standard gamble is without utility in eliciting participants’ values; two of our participants were able to carry out the exercise without difficulty, although not necessarily with a full understanding of its purpose. Also, we are not able to simply attribute difficulty using the standard gamble exercise to elicit values to a deficiency on the part of participants; these participants had already successfully negotiated lengthy and complex information and consent processes required to take part in the two studies. It may be that the standard gamble could form part of an array of processes for eliciting participants’ values. We were able to demonstrate here the value of the qualitative components of the qualitative study to scrutinise methodological validity and in enabling a decision to act ethically in the context of the trial.

Qualitative methods in RCTs

That qualitative research can fulfil the roles outlined in the MRC guidelines, specifically as part of pre-intervention development and post hoc interpretation, is well established.^15–17 A key advantage of integrating qualitative research, particularly qualitative process evaluation, in RCTs is that they allow us to examine social processes and practices engaged in, for example, the use of new technologies^18,19 and thereby gain an understanding about the production of knowledge in particular RCTs. In this study, we have demonstrated the manner in which qualitative process evaluation may contribute to monitoring and auditing trial conduct, and ensuring felicity to the principle of non-maleficence, in parallel and complementary ways to data monitoring committees of RCTs. Further, qualitative methods can be used to evaluate the validity of measures used in a trial in situ—that is, in the local and particular circumstances of particular trials. Notwithstanding these benefits, the incorporation of qualitative research methods may have an adverse effect on recruitment to the trial where additional commitments are required from participants. Any such selection bias would affect the interpretation of the trial, particularly its generalisability. The question of whether a selection bias is introduced by additional qualitative components in a trial requires further examination.

CONCLUSION

We have demonstrated here the use of qualitative methods in examining the processes of a trial, in raising methodological and ethical questions about the conduct of the trial and in providing evidence to take action on those questions where necessary. Although the role of qualitative process evaluation to raise methodological and ethical issues has been discussed elsewhere,²⁰ this is the first demonstration of the use of qualitative evidence to take action to discontinue an intervention arm of a trial. We argue that, in addition to acting as a pre and post hoc adjunct to improving and understanding trials of complex interventions, qualitative evidence can inform decision making about the continuation or cessation of a trial or its component interventions.