Article Text

PDF

What matters to patients? A systematic review of preferences for medication-associated outcomes in mental disorders
  1. Øystein Eiring1,2,
  2. Brynjar Fowels Landmark3,
  3. Endre Aas3,
  4. Glenn Salkeld4,
  5. Magne Nylenna1,2,
  6. Kari Nytrøen1,3
  1. 1Faculty of Medicine, University of Oslo, Oslo, Norway
  2. 2Norwegian Knowledge Centre for the Health Services, Oslo, Norway
  3. 3Department of Quality and Patient Safety, Innlandet Hospital Trust, Brumunddal, Norway
  4. 4University of Sydney, School of Public Health, Sydney, Australia
  1. Correspondence to Dr Kari Nytrøen; kari.nytroen{at}medisin.uio.no

Abstract

Objective To investigate patients’ preferences for outcomes associated with psychoactive medications.

Setting/design Systematic review of stated preference studies. No settings restrictions were applied.

Participants/eligibility criteria We included studies containing quantitative data regarding the relative value adults with mental disorders place on treatment outcomes. Studies with high risk of bias were excluded.

Primary and secondary outcome measures We restricted the scope of our review to preferences for outcomes, including the consequences from, attributes of, and health states associated with particular medications or medication classes, and process outcomes.

Results After reviewing 11 215 citations, 16 studies were included in the systematic review. These studies reported the stated preferences from patients with schizophrenia (n=9), depression (n=4), bipolar disorder (n=2) and attention deficit hyperactive disorder (n=1). The median sample size was 81. Side effects and symptom outcomes outnumbered functioning and process outcomes. Severe disease and hospitalisation were reported to be least desirable. Patients with schizophrenia tended to value disease states as higher and side effects as lower, compared to other stakeholder groups. In depression, the ability to cope with activities was found to be more important than a depressed mood, per se. Patient preferences could not consistently be predicted from demographic or disease variables. Only a limited number of potentially important outcomes had been investigated. Benefits to patients were not part of the purpose in 9 of the 16 studies, and in 10 studies patients were not involved when the outcomes to present were selected.

Conclusions Insufficient evidence exists on the relative value patients with mental disorders place on medication-associated outcomes. To increase patient-centredness in decisions involving psychoactive drugs, further research—with outcomes elicited from patients, and for a larger number of conditions—should be undertaken.

Trial registration number PROSPERO CRD42013005685.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This is the first systematic review on patients’ relative, stated preferences for outcomes of psychopharmacological treatments across methods and disorders.

  • We summarised patients’ preferences for hypothetical outcomes associated with medications and excluded preferences for specific medications or treatment domains, which are amenable to misconceptions. The treatments per se do not give value to the user; it is their outcomes that give value.

  • We tested and applied a broad, peer reviewed search strategy, but we might have overlooked or missed studies. Study quality was rigorously and comprehensively assessed.

  • Owing to the heterogeneity of methods and outcomes we could not perform quantitative summaries of the relative strengths of preferences.

Introduction

To respect and respond to patient preferences is a crucial aim in patient-centred healthcare1–3 and a persistent ideal in evidence-based medicine,4 clinical practice guidelines5 and shared decision-making.6 Integrating patient preferences is increasingly advocated in health technology assessments,7 drug development8 and market approval and reimbursement.9 Yet to allow patient preferences to guide healthcare decisions remains to become common practice.10

In the mental health field, healthcare decisions frequently involve medications: one in five Americans and one in eight western Europeans are prescribed psychotropic drugs.11 ,12 The psychopharmacological dilemmas faced by clinicians and patients are often preference-sensitive, and involve trade-offs of conflicting, multiple outcomes.13–15 Systematic reviews and meta-analyses have summarised whether patients prefer pharmacological or psychological treatment,16 and the effect of matching the treatment to the patient's preferred option.17 ,18 However, in trade-off dilemmas, studies on patients’ preferred options might be less informative than studies on their preferences for the outcomes of the options. Knowledge about the relative strengths of preferences for treatment outcomes, representative for populations, can be gained with stated preference methods. A range of techniques is available.19–21 Systematic reviews of studies applying the techniques to elicit patient preferences for outcomes of psychotropic drugs are lacking. The current void of knowledge on the outcomes patients value the most and least, and what those outcomes should be,22 strikes the foundation of patient-centred care and suggests missed opportunities for more patient-centred decisions.

For these reasons we conducted a systematic review of studies on patients’ valuations of outcomes associated with psychoactive medications. The main goal was to summarise what is known on the relative strengths of preferences. We also reviewed:

  • Whether patient perspectives were part of the purpose and construction of outcomes

  • Which outcomes were addressed

  • The feasibility of stated preference methods for patients with mental disorders

  • Correlations between patient preferences and demographic or disease variables

  • Differences between patients’ preferences and those of other stakeholders

Methods

This study followed the PRISMA reporting guidelines (see online supplementary appendix 10): http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000097

Eligibility criteria

Studies applying stated preference methods to elicit the relative values patients place on outcomes of psychopharmacological treatments, using quantitative methods, were eligible for inclusion. Studies that included rating scales only were excluded due to doubts about whether the techniques adequately measure strength of preference.19 No publication date, context or publication status restrictions were imposed.

We included studies on adult patients with direct experience of the mental disorder specified in the study, currently diagnosed with or at risk of recurrence of the disease. Trials addressing patients with substance-related and addictive disorders were excluded.

Patient preferences can be defined as “statements made by individuals regarding the relative desirability of a range of health experiences, treatment options, or health states”.23 We restricted the scope of our review to preferences for outcomes, including the consequences from, attributes of, and health states associated with particular medications or medication classes and process outcomes. Studies on patients’ preferences for specific medications, medication classes, or treatment domains, or for health states detached from a medication context, were not included. Studies measuring health-related quality of life were excluded unless they elicited patients’ relative valuations of outcomes.24 Studies calculating preferences by mapping life quality scales to utility scores were not included. Owing to the heterogeneity of the field, we did not specify the outcome measures in detail before the study.

Search strategy

We searched Medline, Embase, PsycINFO, CENTRAL, SveMed+, The Health Technology Assessment Database, the NHS Economic Evaluation Database and grey literature databases from inception to September 2013. We piloted our strategies in a test search and modified the use of keywords and indexed terms. A PRESS (Peer Review of Electronic Search Strategies) review was undertaken and the strategies revised. In the revised search, we used a combination of subject headings, subheadings and text words. The bibliographies of the included studies were hand searched for additional studies. Online supplementary appendix 1 details the search strategies.

Study selection

Two review authors (KN, BFL) independently reviewed all identified titles and abstracts (figure 1). Full text articles were obtained for potentially relevant trials and examined in detail by the same authors. Disagreements were discussed with the principal author (OE) and resolved by consensus.

Figure 1

Flow chart of study selection.

Quality assessment

Standardised criteria for methodological quality and risk of bias in stated preference studies have not been established.25 ,26 To enable critical appraisal, we constructed a checklist based on criteria proposed for assessment of stated preference research in eight methodological reviews and evaluation tools.19 ,27–33 The resulting inventory consisted of 31 quality criteria within five domains: external validity, presentation of outcomes, minimisation of irrelevant variance, reporting and analysis, and other aspects (see online supplementary appendix 2). Two authors (KN, OE) independently assessed all studies considered for inclusion on the 31 items. Studies given an overall intermediate or high quality rating were included. Inconsistencies were resolved by discussion.

Data collection

We developed, piloted and revised a data extraction form consistent with the goals of the review. Two reviewers (KN, OE) independently extracted data on the study, study population, preference elicitation aspects and preference results (see online supplementary appendix 3).

Results

Of 11 215 unique citations, 54 proceeded to full text review. We excluded 39 studies, the most frequent reason being lack of quantitative data on the relative strengths of the preferences (see online supplementary appendix 4). This left 16 studies for our systematic review. We present the results in descriptive and tabular form.

Study characteristics

Sixteen studies in 16 papers included 1785 patients with a median sample size of 81 (range 20–469). Nine studies had assigned patients with schizophrenic disorders, four had depressive disorders, two had bipolar disorders and one had ADHD (attention deficit hyperactive disorder). The range of reported mean ages was 39–46 and the median percentage of female participants 55. In the seven studies reporting ethnicity, the median percentage of Caucasians was 86. Ten studies included only outpatients, one inpatients and outpatients, and five did not report hospitalisation status. Twelve of the 16 studies were partly or fully conducted in the USA. Preference elicitation was the main or part of the main objective in 15 studies. Preferences were elicited from patients with the standard gamble (SG) method in six studies, conjoint analysis (CA) and pair-wise comparison (PC) in two studies each, discrete choice experiment (DCE) and willingness-to-pay (WTP) in two, and time trade-off (TTO) in one (table 1). Basic descriptions of the methods are provided in box 1.

Box 1

Stated preference methods in healthcare

  • The standard gamble elicits the value of outcomes by asking patients to choose between a certain outcome and a gamble.

  • Willingness to pay is the maximum amount a patient is willing to offer to obtain good, or to avoid undesirable, outcomes.

  • In conjoint analysis, patients place weights on different features of a health option.

  • In pairwise comparison, patients compare health options in pairs to find which is preferred, or which has the largest amount of a measurable aspect.

  • In discrete choice experiments, patients state their preference over alternative scenarios, such as health states.

  • In time trade-off, patients are asked to choose between living in a suboptimal state for a certain period of time, versus living a healthier life for a shorter time.

Table 1

Description of studies included in the systematic review

Study quality

A frequently used quality criterion for stated preference studies is whether the outcomes are presented to the participants in adequate detail.28 The level of detail varied in the included studies, from short text descriptions to info-graphics and videos with actors enacting symptoms and side effects. All included studies minimised threats on validity from factors irrelevant to the represented outcomes using measures such as precomprehension or postcomprehension tests.

Non-random recruitment procedures limited the external validity in all studies. Four studies included less than 50 participants. In six studies, data were not reported for participants who did not complete the procedures. Authors generally provided incomplete information on study design, and five studies lacked measures of variance. The use of statistical techniques was deemed appropriate in all studies. Only two studies were given an overall ‘high quality’ rating (see online supplementary appendix 5).

Purposes

Seven studies related their results to potential benefits for individual patients, for instance to tailor adherence programmes,34 to be helpful in medical decision-making35 or to promote concordance between patients and psychiatrists.36 Although 13 of the studies received funding from a pharmaceutical company, only five studies suggested or performed economic analyses from an industry perspective.37–41 Eight studies discussed how their preference results could be used in public evaluation and prioritisation contexts (see online supplementary appendix 6).

Outcome sources

In 10 of the 16 studies, input from patients was not sought when outcomes were selected and constructed. The six studies that obtained patients’ perspectives used interviews,35 ,40 focus groups,36 ,42 piloting of the suggested outcomes39 ,43 or comments on patient websites.40 In comparison, six studies engaged clinicians and 13 studies reported using research or literature to identify the outcomes. Three used research or literature only38 ,44 ,45 (see online supplementary appendix 6).

Included outcomes

Side effects (n=14) and symptoms including relapse (n=11) were most frequently presented to participants. Six studies included process related outcomes, for instance, the treatment schedule, or number of visits to the hospital. Functioning featured in six studies and costs in three. While a large number of outcomes are potentially relevant,25 only a limited number was presented in the studies. For example, mortality or burden to relatives was not included in any study (see online supplementary appendix 6 and 9).

Feasibility and validity

The studies reported that patients were able to provide usable preference measures for the six methods applied, generally comprehended the tasks and gave sufficiently consistent answers. A total of 92–100% of the participants completed the procedures.

Three of the nine studies with patients with schizophrenia reported moderate or major problems. In the first, a small SG study, 30% of the patients did not understand the survey well and 56% had inconsistent rank ordering.46 In the second, also a SG study, patients made more logical errors than others and mostly, in contrast to other stakeholders, preferred not to correct their mistakes.28 In one of the two studies applying conjoint analysis, patients reported lower levels of understanding and more difficulty with the task compared to other participants.45 Minor or no problems were reported in the two schizophrenia studies applying TTO and self-explicated methods.

The studies including patients with depression, bipolar disorder and ADHD reported minor or no feasibility problems, but feasibility and validity were less focused on compared to the schizophrenia studies (see online supplementary appendix 7).

Correlations with patient characteristics

Eleven studies investigated whether patient preferences correlated with demographic or disease variables, with negative or conflicting findings.

Three studies found that preferences correlated with age,35 ,37 ,41 whereas five found no significant association.38 ,40 ,42 ,43 ,47 Gender correlated with preferences in one study,41 but did not correlate in four.35 ,38 ,40 ,47 Possible correlations with living arrangement, education, employment-status and income level were investigated with negative or mixed results.

Severity of disease correlated with preferences for hypothetical health states43 ,47 and the impact of a side effect on utility.28 Two studies40 ,42 found that disease severity did not correlate with preferences, and one study reported mixed results.43

Comparison with other stakeholder groups

Eight studies, all on schizophrenia, compared the preferences of patients with those of other stakeholder groups.28 ,36 ,39 ,40 ,44–46 ,48 Patients’ preference values differed systematically from those of other stakeholders in the five studies published after 1997, and the magnitude of the differences varied from modest to considerable.28 ,36 ,40 ,44 ,45 People with schizophrenia valued disease states higher than other stakeholder groups did.28 ,40 ,44 Extrapyramidal side effects (EPS) were given a lower value or deemed more important, compared to clinicians,28 ,44 ,45 except in one study.48 The preferences of family members were closer to those of patients, compared to psychiatrists and laypeople, in studies that performed relevant comparisons.28 ,44 ,48 Other stakeholders did not value functioning or symptoms significantly and/or consistently different from what patients did (see online supplementary appendix 8).

Strengths of preferences

Stated preference methods elicit different preference measures. WTP represents the value that an individual places on a commodity, SG and TTO estimate a utility, most often on a scale where 0 is death and 1 perfect health. CA and DCE measure the relative importance or value of different outcomes.19

Schizophrenia

‘Positive’, ‘acute’ or ‘psychotic’ symptoms figured consistently among the least desirable outcomes to patients.36 ,40 ,45 Negative symptoms such as reduced capacity for emotion were found more desirable or less important than positive symptoms.36 ,45 ,48

Independency was rated highly,36 ,45 and being an inpatient, lowly.48 Cognitive36 ,42 and social36 ,42 ,45 functions were moderately or highly important compared to other outcomes. The importance of capacity for work and for daily living was intermediate.36 ,42.45

EPS was included in seven studies. Two small studies39 ,46 both reported that the disutility of parkinsonism was larger than the disutility of akathisia and tardive dyskinesia. The presence of EPS reduced the utility by 12–21%. Pseudoparkinsonism reduced the utility with 5–7% in two other studies.28 ,44 In three additional studies the relative importance of EPS was moderate or high, compared to other outcomes.40 ,42 ,45 Health states with weight gain had a higher utility than states with EPS in the only schizophrenia study including side effects other than EPS.40

Depression

Severe, untreated depression reduced the utility from 1.0 (perfect health) to 0.3, and 25% of patients considered this state equivalent to or worse than death.47 Patients’ reduced ability to start and cope with activities on their own, due to fatigue, was more important than depressed mood in one large, well-performed study.35 The same study found that side effects after 2 weeks also were more important than depressed mood.35

The simultaneous presence of weight gain and no orgasm reduced the WTP from USD 686 per month for an antidepressant without side effects, to USD 227.41 One study found very small differences in side effect utilities.47 Patients were willing to pay more to avoid tremor and sleepiness than to avoid dry mouth and sweating, according to one study.38

Bipolar disorder

The inpatient state, inpatient mania and severe depression had lower utilities than the outpatient, stable state in one study.43 The relative strengths of preferences for mania versus depression were conflicting in two studies.34 ,43 Cognitive effect and severity of depression were equally important in one study.34

Weight gain within 3 months was found to be equally important to cognitive impairment and severity of depression, and three times more important than serious side effects34 A weight gain of more than 2.3 kg reduced the utility with 0.07.43

ADHD

Patients were willing to pay 74% more for functioning well in the morning and school/workday, compared to functioning well in the afternoon/evening in one study.37

Table 2 and online supplementary appendix 9 contain additional details on all the conditions.

Table 2

Relative strengths of preferences

Discussion

Principal findings

Benefits to patients and clinical practice were part of the purpose in a minority of the 16 studies included in this review. Most authors had not involved patients when they selected and developed the outcomes in their studies. Side effect and symptom outcomes outnumbered functioning and process outcomes, and only a limited selection of potentially important outcomes were presented to patients. The stated preference methods were generally found to be feasible across different conditions and disease severities, but patients with schizophrenia experienced more problems with the tasks than other patient groups, in particular for SG. The patients’ preferences did not vary consistently with age, gender, disease severity or other demographic or disease variables. The relative preferences of patients with schizophrenia differed systematically from those of other stakeholders in most studies. Patients valued disease states higher than did other groups and perceived side effects more negatively than clinicians did. Patients with schizophrenia desired acute and psychotic symptoms least of all outcomes, and valued independency highly. Functioning occupied a middle ground; social function tended to be more important than vocational function. The importance of EPS was moderate or high. For patients with depression, severe disease greatly reduced utility, though the ability to cope with activities, and presence of side effects, appeared more important than a depressed mood, per se. Patients with bipolar disorder valued inpatient mania and severe depression lowly, and reported weight gain to be important. In ADHD, patients reported that functioning in the morning and during daytime was most important.

Strengths and weaknesses of the review

This is the first systematic review on patients’ relative, stated preferences for outcomes of psychopharmacological treatments across methods and disorders. The review accords with the PRISMA guidelines (see online supplementary appendix 10) and the protocol was registered in the PROSPERO database prior to conduction (see online supplementary appendix 11).

We summarised patients’ preferences for outcomes associated with medications and excluded preferences for specific medications or treatment domains, which are amenable to misconceptions. The treatments do not give value to the user, per se; it is their outcomes that give value.49 In a trade-off situation, the best option reflects the partial valuations and probabilities of those outcomes.50

Preference studies are not archived in reference databases with common subject headings and keywords that allow a highly sensitive and specific retrieval.25 We tested and applied a broad, peer reviewed search strategy, but we might have overlooked or missed studies.

We rigorously and comprehensively assessed study quality. Several included studies risked multiple biases and stricter criteria could have been applied. However, quality criteria for preference studies can conflict and methodological rigour must be balanced with the cognitive effort demanded from participants.

Owing to the heterogeneity of methods and outcomes, we could not perform quantitative summaries of the relative strengths of preferences, as is common in systematic reviews on preference studies.26 ,51 ,52 Our review includes studies that were not powered to provide statistically significant differences for strengths of preferences. Preferences for outcomes were elicited from varying and often small numbers of participants, with heterogeneous disorders.

The end of the search period was September 2013, thus at the time of publication the search is relatively old in comparison to the median for systematic reviews.53 ,54

Stated preference studies elicit preferences in hypothetical choices, and the outcome preferences in a real setting might be different. The reliability and validity of specific patient preference methods is debated, and the techniques and quality standards are likely to change in the future.21 ,55 ,56

Results in context

The call for outcomes that are meaningful and important to patients is increasing57 ,58 Patient-centred outcomes are often contrasted to clinical outcomes such as symptom control and side effects. They assess the impact of illness and therapy from patients’ perspectives, and should be those that patients notice and care about.59

In contrast to this aim, we found that outcomes presented for preference elicitation were mostly selected and described without input from patients. Other systematic reviews on stated preferences confirm this finding. In a review of experiences of healthcare delivery, few outcomes were worded from patients’ perspectives.21 In a review on diabetes care, only 3 of 14 studies had employed focus groups in the outcome selection process.52 Disease-specific reviews often do not report on patient involvement.26 ,51 ,60 In line with this lack of patient-centredness and similar to our findings, symptom outcomes and adverse effects are most frequently included in preference studies, at the expense of other outcomes.25

Suggested patient-centred outcomes in mental disorders include social and vocational functioning, body image, reduced stigma, recovery and reduced burden to caregivers.22 In schizophrenia alone, 194 non-traditional outcomes have been suggested.61 We found that side effects and in particular EPS and weight gain were important to patients, both relative to other outcomes and to other stakeholders. Side effects have an impact on health status, but to patients, their effect on physical attractiveness and the associated status, self-esteem and social opportunity might be more important.62 When the functional consequences of adverse events and no treatment are similar, people value avoiding the adverse effects most.51

In addition to side effects, severe symptoms were highly important to patients, whereas functioning was moderately important. The claim that patients value functioning higher than symptom-oriented, ‘textbook’ outcomes, was therefore not supported.36

We found significant differences in how patients and non-patients valued outcomes. This topic is currently debated. Accordant with our findings, the most comprehensive meta-analysis to date63 concluded that patients value health states higher than the general public. The difference was small to moderate, and notably the valuations differed less when both groups valued descriptions of health states, instead of patients valuing their actual health. A possible explanation for a difference is that people adapt when they become ill: we develop skills, adjust activities and expectations, and redefine what is good health and a good life.55 ,64 ,65 Notably, different valuations of one-dimensional health states do not necessarily translate to differing partial utilities of health states and to process outcomes.66

Concerns have been raised that cognitive impairment, limited self-insight and distortions of reality impede patients’ use of stated preferences methods, and could leave the results meaningless, in particular in schizophrenia.67 ,68 The results of the studies in this review indicate that several stated preference methods might be feasible for identifying relative outcome preferences from patients with mental disorders, and that validity is comparable to other stakeholder groups. In a systematic review, the practicality of TTO and CV (contingent valuation) was found to be generally good in patients with depression and schizophrenia. Two of the four studies in the review reported that patients with schizophrenia had some difficulties with the SG tasks.69 The need to improve the techniques persists.70

Meaning of study

Patients report that being told the risks and benefits of treatments is one of the 10 most important aspects of healthcare.71 However, which risks and benefits clinicians communicate to patients is a matter of choice. This review highlights outcomes and outcome priorities clinicians should consider bringing into their conversations with patients when they discuss and decide between psychotropic drug options.

Our findings could inform on the benefits and harms to include in patient decision aids, which are tools designed to help patients participate in making choices among healthcare options.72

It has been suggested that authors of clinical guidelines should conduct a systematic review on patient preferences in the relevant content area.28 ,51 The stated preferences presented in this review could be used as an early point of reference when guidelines are developed for schizophrenia, depression, bipolar disorder and ADHD.

In situations where stakeholder groups have different values, spotlighted in this review, a main question is whose preferences should be accommodated? Proponents of the patients’ preferences argue that people with the relevant disease are the best judges of their own welfare, and that true preferences require experience with the outcome. Opponents claim that the judgments of non-patients are more appropriate, because decisions affecting resource allocation for one group of patients affect the provision for other groups.55 The dilemma is not empirical, but normative, and the answer depends on the decision context.55 ,73

In economic assessments, the relative preferences from patients reported in this review might inform regulatory benefit-risk assessments and be used in direct comparisons of drugs,9 but in this field, the preferences of the general population is currently the norm.74

Unanswered questions and implications for research

Although many studies have addressed ‘what matters’ to patients with mental disorders, few have investigated the relative preferences for medication outcomes. Current knowledge is fragmented and exists for a limited number of aspects and conditions only. Surprisingly, only a minority of the studies have been performed from patients’ perspectives. The evidence does not allow firm conclusions on what outcomes of psychotropic medications matter most to patients, and there is an obvious need for more research.

Insufficient reporting in stated preference studies is widespread.51 ,52 ,60 Concise reporting of all study dimensions, including variance, study design and the outcome descriptions presented to patients, is necessary. Although the studies in this review generally found that stated preference methods were feasible for patients with mental health disorders, challenges were also exposed, demonstrating that the techniques need to be improved and tailored to the relevant populations.

Conclusion

Despite the widely declared prominence of patient preferences in healthcare, knowledge on which medication-related outcomes matter most to patients with mental health disorders has been largely absent. Clinicians and policymakers should be aware that patients’ priorities might be different from theirs and that they cannot reliably be inferred from patients’ demographic characteristics or health status. To improve health outcomes for patients, we need more evidence on the relative importance patients place on relevant outcomes. In clinical practice, knowledge on group-level preferences can be a starting point, but to know what matters most to the person in front of you, you have to ask.

Acknowledgments

The authors acknowledge the helpful insights provided by Jack Dowie.

References

View Abstract

Footnotes

  • Contributors OE was involved in study design, systematic search, pilot of data extraction form, title/abstract scanning, determining eligibility of articles, quality assessment of articles, data extraction, data synthesis, data interpretation, literature search, and writing and revising paper and appendices. BFL was involved in title/abstract scanning, determining eligibility of articles, obtaining full text and revising manuscript. EA was involved in systematic search, writing the article and see online supplementary appendix 2. MN was involved in study design, data synthesis, data interpretation and revising the manuscript. GS was involved in study design, determining eligibility of articles, quality checks of included articles and appendices, and revising the paper. KN was involved in study design, title/abstract scanning, pilot of data extraction form, determining eligibility of articles, quality assessment of articles, data extraction, data synthesis/analysis, data interpretation, literature search, and writing and revising the paper and appendices. All authors had full access to the data, approved the final draft, and take responsibility for the accuracy of the analysis and the integrity of the data. OE and KN are guarantors.

  • Funding This study was funded by the Norwegian Research Council, the Norwegian Knowledge Centre for the Health Services, and the Sykehuset Innlandet Hospital Trust, Norway. The work was undertaken by the Evicare (Evidence-based care processes) research and innovation project partnership (http://www.kunnskapssenteret.no/prosjekter/evicare-evidence-based-care-processes-integrating-knowledge-in-clinical-information-systems).

  • Competing interests All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement Full data sets and technical appendices are available with open access from corresponding authors.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.