Vignette experiment

Vignettes are hypothetical scenarios that mimic real-life situations, enabling researchers to evaluate their phenomenon of interest in a controlled experimental setting.13 Since there is little noise in a designed experiment, covariates can be assessed in a relatively unbiased manner.13 Additionally, the use of vignettes precludes the need to recruit a balanced sample of subjects, as such a sample can simply be simulated. In our case, we aim to establish what drives the decision-making process surrounding secondary prophylaxis of VTE. In a real-life setting, it would be difficult to recruit a diverse enough set of patients to take the various risk factors that may influence this process into account. With the vignette experiment described in this paper, we achieved a highly efficient alternative to a much more costly and time-consuming real-life experiment.

Preselection of attributes

From December 2015 to February 2016, we conducted 13 semistructured interviews with an equal number of senior-level physicians affiliated with medical centres around the world that participate in the Einstein Choice study.14 The purpose of these interviews was to establish which factors are particularly important in determining the duration of secondary prophylaxis of VTE, and, conversely, which factors weigh against a decision to continue treatment past the resolution of the initial event. Our focus during these interviews was on specialist decision-making in the period directly following the resolution of the initial event, and so patient’s opinion and risk factors relating to the period following the discontinuation of treatment (eg, D-dimer levels) were not taken into account. Our questions were informed by a review of the pertinent literature and guidelines,11 12 which had yielded an initial selection of relevant attributes. Importantly, we decided to exclude cancer as a risk factor, as its impact on the decision to continue or stop anticoagulation is too dominant: presence of an active cancer invariably warrants indefinite anticoagulation, regardless of other factors.11

In our interviews, we specifically asked our interviewers to identify patient characteristics that significantly impact either the risk of VTE recurrence or the risk of bleeding. Although all physicians weighted individual risk factors differently, common factors and their corresponding levels were identified. table 1 is a frequency table of the risk factors that were considered to significantly impact either on VTE recurrence risk or on bleeding risk, by number of physicians who mentioned them. This table does not include risk factors that were mentioned by fewer than two interviewees.

Reducing design complexity

The preselection of attributes yielded 18 factors that were indicated to be of importance by at least two physicians. A recommendation for conjoint experiments, which exhibit commonalities with our approach, states to limit the complexity of a design as much as possible, as an overly complex design may confuse respondents and distort the responses to an experiment.15 We first discarded factors that were unsuitable for the vignette experiment due to their ungeneralisability: uncontrolled hypertension and physical activity. We then removed the ‘unstable or high INR’ factor on account of it being specific to vitamin K antagonists. We further trimmed down the list of factors by discarding the risk factors that only rarely occur in real-life patients with VTE: thrombocytopenia (<1%)16 and liver disease (~1.5%).17 Finally, we combined ‘signs of PTS’ with the ‘type of VTE’ factor, so as to prevent inappropriate interactions: post-thrombotic syndrome almost exclusively occurs in tandem with DVT, rather than with PE.18 This leaves 12 factors in our reduced model.

Experimental design

Given 12 factors, of which nine have two levels; two have three levels; and one has six levels, there is a total of 2⁹3²6=27 648 level permutations in the full factorial model. As having respondents evaluate all 27 648 possible scenarios would be prohibitively costly, we created an efficient fractional factorial design via Fedorov’s 1972 exchange algorithm.19 To generate this design, we utilised the R package ‘AlgDesign’,20 an implementation of the Fedorov algorithm. The Fedorov algorithm randomly selects rows from a full factorial matrix X to create a new matrix ξ, and then exchanges rows from ξ with rows from X\ξ to optimise a specified criterion. In our case, we optimised our reduced design for D-efficiency and G-efficiency.

The number of vignettes in the reduced design was selected on the basis of the following criteria:

1. D-efficiency

2. G-efficiency

3. Level balance

4. Feasibility of the required sample size

Number of vignettes in the reduced design

Based on the above considerations, we selected a design ξ with N_ξ  = 72. This number of vignettes yielded a well-rounded balance between our efficiency criteria, as illustrated in figure 1 and table 2. To reduce workload for our respondents, we divided this design into six blocks of 12 vignettes each, using the optBlock function in the ‘AlgDesign’ R package. After optimisation, we achieved a block design diagonality of 0.88. Plugging in the values for a single block into our sample size formula returns a required sample size of . As there is considerable overlap between blocks, we assume a sample size of 250 to be sufficient to draw reliable inferences with the design as specified above. Following the ‘six-times rule’, accounting for twelve vignette evaluations per respondent, yields a required sample .

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

D and G efficiencies of design matrix ξ.

Confirmation of the required sample size via a priori power analysis

As we intend to analyse the experimental results with mixed-effects models (see Analysis of the results section), it is necessary to validate the sufficiency of the above sample size calculation.30 We performed a series of simulations, using the R command ‘simulate’ and the ‘makeLmer’ and ‘makeGlmer’ commands in the R package ‘simr’,31 to construct an a priori power analysis. We conducted two linear mixed effects models and four generalised linear mixed effects models with logit link functions using the R package ‘lme4’.32 We used pilot data (see Pilot evaluation of the design section) to construct these models. Specifically, we regressed outcome variables ‘VTE recurrence risk’ and ‘bleeding risk’ (see Outcome variables section) on a vector of covariates, consisting of the selected patient risk factors and participating specialist characteristics (see Physician characteristics section). We then systematically reduced these models via AIC-based forward and backward stepwise variable selection, and by eliminating variables with clinically insignificant effect sizes (<|0.01| on a scale 0–10) or particularly unpromising p values (>0.6), as determined via Type II Wald X² tests. We created four binomial generalised linear mixed models to separately analyse the four treatment outcomes in our original pilot survey (continue 3 months; continue 6–12 months; continue indefinitely; cease treatment or switch to aspirin). For the ‘continue’ treatment options we adapted and modified the ‘recurrence risk’ model’s equation until it exhibited adequate fit and model convergence. For the ‘stop treatment’ option we used the same process, this time modifying the ‘bleeding risk’ model’s equation. Random slopes and intercepts were included as much as possible; the final selection of mixed effects was based on model improvement and convergence. The recurrence risk model includes 10 covariates, one random slope and three random intercepts. The bleeding risk model features 7 covariates, two random slopes and three random intercepts. The model corresponding to treatment 1 features 8 covariates and a random intercept for participants. Logistic model 2 includes 5 covariates and three random intercepts (participant, block identifier, country). Logistic model 3 incorporates 7 covariates and random intercepts for participant and block identifier. Logistic model 4 features 6 covariates and three random intercepts (participant, block identifier, country).

From these models created on the basis of pilot data, we extracted the covariate coefficients , the variance and correlation components and the residual SD . We then generated covariate (X) matrices corresponding to varying sample sizes by concatenating blocks of vignettes and simulating plausible participant characteristics. Finally, we simulated responses for each of the models at variable sample sizes. We defined the power of all models as the proportion of incorporated covariates with p<0.05, assessed with Type II Wald X² tests. The power analysis is summarised in table 3 and figure 2. The power analyses indicate that the lower bound of the sample size should be adjusted to n=100. The original upper bound of n=250 was validated as amply sufficient to estimate coefficients with reliable power.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Power by sample size.

Pilot evaluation of the design

In May 2016, we sent out Microsoft Word-based pilot surveys to 12 thrombosis experts to evaluate our initial design. From the comments we received, it was apparent that some of the factors and levels in our model were not formulated precisely enough. For instance, pilot participants indicated that in patients who had had a previous VTE, it was important to know when this had occurred. For this item, we added a timeframe to improve clarity. Some participants struggled with the unqualified formulation ‘renal function: insufficient’. We included the more detailed description ‘<50 mL/min’ in these cases, referring to the corresponding creatinine clearance rate. Additionally, of patients who received concurrent antiplatelet therapy, a number of pilot participants said to be confused as to whether the practitioner treating the patient for VTE recurrence could cease this concurrent therapy. As our intention was to refer to patients with an absolute indication for antiplatelet therapy, we reformulated this item to more explicitly refer to such patients. With these changes, we arrived at our final design, shown in table 4.

Physician characteristics

From our expert interviews, it was apparent that physicians involved in the secondary prevention of VTE all have unique perspectives on thrombosis and bleeding risk in VTE. In order to understand why physicians who are faced with the same clinical vignettes may respond to these differently, we decided to include a number of questions relating to characteristics of the physicians themselves. Before commencing the experiment, physicians are asked to provide their age, gender, country of employment, specialty, years of experience with the treatment of VTE, and an estimate of the number of unique patients with VTE they treat per annum. These physician characteristics were selected on the basis of previous research into physician guideline adherence and clinical decision-making.33–37

By incorporating the ‘country of employment’ variable as a random effect, we hope to capture the variance resulting from any country-specific aspects of physician’s decision-making that would otherwise be difficult to quantify.

We provided six possible answers to choose from in response to the specialty question, that is,: internal medicine, vascular medicine/angiology, cardiology, haematology, pulmonology or other. The specialty of responding physicians will also be included as a random effect in our analysis.

Outcome variables

As outcome variables, we incorporated two continuous variables to respectively estimate VTE recurrence risk and bleeding risk on a 100-points visual analogue scale indicating low to high risk, and five discrete variables: continue treatment for three additional months, continue 9 additional months, continue indefinitely, stop treatment, or switch to aspirin. These choices were based on the most recent CHEST and ESC VTE treatment guidelines,11 12 the conducted expert interviews and the comments received on our pilot design.

Implementation of the survey

The survey has been implemented online by independent web developers and is currently live at www.vte-survey.com (figure 3). Survey respondents are randomised to one of six pre-specified blocks at the start of the experiment.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Screen capture of the web-based survey (www.vte-survey.com).

As mentioned above, one aspect of our experimental design was to make the workload for study participants as slight as possible. For this reason, in the online implementation of the survey the progress of participants is continuously recorded. This allows participants to take breaks during the survey and even to leave the website, without their responses up until that point being forfeited. Participants may return to the website at any time and resume the survey where they left off. In our current sample of respondents, there is a small number of such cases that skews the mean survey completion time considerably upward. A better reflection of the average workload is given by the median survey completion time, which in our current sample is equal to 9 min. Based on this number, we conclude that the risk of participant fatigue is negligible for our study.

Data collection was commenced in July 2016, and is still ongoing. We have reached our stated sample size requirements in January 2017, but will leave the website live for at least the remainder of this year, so that more data may be recorded for future analysis. We welcome any specialists involved in the preventive treatment of VTE to participate in this experiment.

Participant recruitment strategy

As we are interested in specialist decision-making, our recruitment strategy focuses exclusively on specialists. This group of subjects is particularly difficult to reach in a non-systematic or stochastic manner. This section details how we approached the recruitment of participants in a way that we believe maximised the chance of reaching a broad and representative sample.

In order to meet the sample size requirements, as determined by the vignette-based sample size formulae as well as the power analysis described above, we have leveraged several channels to recruit study participants so far. First, we contacted a list of several hundred specialists who participate or have participated in the Einstein Choice study.14 Participation in this context is defined as having recruited at least one patient for the Einstein Choice study, and the list features specialists of variable ages, professional stature and levels of experience. We also contacted professional collectives of specialists, such as the International Society of Thrombosis and Haemostasis (ISTH), the ESC and the Asian Pacific Society on Thrombosis and Hemostasis (APSTH), which led to calls for participation in the ISTH (October 2016) and APSTH newsletters (January 2017) and on their respective websites, and a call for participation on the youth community (EAPC) of the ESC on LinkedIn. Finally, we asked each specialist who responded favourably to our request to participate to disseminate the survey among colleagues and residents.

Analysis of the results

We will conduct several separate analyses of our results (figure 4). A first linear mixed effects regression model will relate all risk factors and physician characteristics to the continuous estimated VTE recurrence risk outcome variable. A second model does the same for bleeding risk. After having estimated the coefficients in these two linear models, we will use generalised linear mixed models (GLMMs) to estimate the ORs for all independent variables, including physician characteristics, in relation to the choice of treatment options. We will use a GLMM with a binomial logit link function for grouped outcomes (stop vs continue treatment), and a multinomial logit regression to discover which variables best predict the five separate treatment outcomes.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Analysis flowchart.

In the linear models, we will evaluate model fit via diagnostic plots (AV plots, Q-Q plots, standardised residuals vs fitted plots), the adjusted R²  and relative measures such as the Akaike Information Criterion (AIC). In the logistic models, we will employ diagnostic plots, the area under the receiver operating characteristic curve (AUROC), McFadden’s pseudo-R² (ρ²) 38 and the predictive accuracy of the models on a test set (kept separate from the development set) to evaluate model fit.

The choice for mixed effects analyses was informed by the theoretical assumption that there is considerable inter-physician variability, and to control for random variance associated with aspects of no interest (eg, block allocation). While analysis of the fixed effects (ie, the regression coefficients) will provide general insights as to which features are of particular interest in determining a patient’s risk of VTE recurrence or bleeding, it is the decomposition of the error term that will show us specifically where and to what extent physicians disagree with one another. To this end, we will specify random slopes that vary by physician for as many relevant features as possible. Our intention in this regard is to keep random effects structures ‘maximal’,39 although within the realistic constraints imposed by the data.30 All analyses will be conducted in the statistical software package R.