Article Text
PDF
Abstract
Objectives To assess the communicative quality of colorectal cancer patient decision aids (DAs) about treatment options, the current systematic review was conducted.
Design Systematic review.
Data sources DAs (published between 2006 and 2019) were identified through academic literature (MEDLINE, Embase, CINAHL, Cochrane Library and PsycINFO) and online sources.
Eligibility criteria DAs were only included if they supported the decision-making process of patients with colon, rectal or colorectal cancer in stages I–III.
Data extraction and synthesis After the search strategy was adapted from similar systematic reviews and checked by a colorectal cancer surgeon, two independent reviewers screened and selected the articles. After initial screening, disagreements were resolved with a third reviewer. The review was conducted in concordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. DAs were assessed using the International Patient Decision Aid Standards (IPDAS) and Communicative Aspects (CA) checklist.
Results In total, 18 DAs were selected. Both the IPDAS and CA checklist revealed that there was a lot of variation in the (communicative) quality of DAs. The findings highlight that (1) personalisation of treatment information in DAs is lacking, (2) outcome probability information is mostly communicated verbally and (3) information in DAs is generally biased towards a specific treatment. Additionally, (4) DAs about colorectal cancer are lengthy and (5) many DAs are not written in plain language.
Conclusions Both instruments (IPDAS and CA) revealed great variation in the (communicative) quality of colorectal cancer DAs. Developers of patient DAs should focus on personalisation techniques and could use both the IPDAS and CA checklist in the developmental process to ensure personalised health communication and facilitate shared decision making in clinical practice.
- gastrointestinal tumours
- colorectal surgery
- gastrointestinal tumours
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information. All reviewed materials were made freely available on the internet by their developers. A full list of the DAs and their titles can be found in table 3. Individual assessments of each reviewer are stored in a secure data repository at Tilburg University Dataverse and is available on request by contacting the main author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first large-scale and comprehensive systematic review on stage I–III decision aids (DAs) for patients with colorectal cancer.
Both academically tested DAs as well as DAs that patients can find online were included to create a more accurate picture of the quality of DAs patients can find and use in clinical practice.
Although the included DAs are freely available to patients, we cannot conclude based on this review whether they are actually being used in clinical practice.
Introduction
Colorectal cancer (CRC) is the third most common cancer in the world.1 With emerging knowledge and availability of technology, the therapeutic options for these patients are increasing. For instance, selected early-stage CRC may now be removed with a minimally invasive endoscopic approach.2 However, these tumours carry a small risk of metastatic spread to the regional lymph nodes which are left behind after endoscopic treatment.3 This risk may be lowered by removal of these lymph nodes but this in turn requires additional surgery with its inherent risks for postoperative complications. Similar considerations come into play with regard to adjuvant chemotherapy after curative resection of high-risk CRC4 and whether or not to treat with neo-adjuvant radiotherapy in rectal cancer.5 6 Recently, a ‘watch-and-wait’ approach is gaining popularity in patients with complete clinical response after radio-chemotherapy as an alternative to radical surgery.7
In all these scenarios, the potential beneficial effect on oncological outcome of a more radical approach should be weighed against the possible negative effects on long-term quality of life.8 To help weigh the pros and cons of these treatment decisions, so-called patient decision aids (DAs) have been developed. Such tools specifically aim to assist patients and clinicians with decision making so that the patient has a better understanding of the treatment options and has insight in their personal preferences regarding treatments. In turn, patient and clinician discuss these personal preferences during consultation, so that they jointly decide which treatment is best. This process is called ‘shared decision making’.9
A recent systematic review assessing the quality of such DAs for various diseases concluded that patients using DAs have (1) a better knowledge of treatments, (2) are better informed about treatments and (3) have a better understanding of their personal values compared to patients in usual care.10 This seems promising, but patients also increasingly use the internet as an important source of health information11 and the DAs found through the web are not included in such large-scale systematic reviews. Therefore, conclusions drawn from such reviews do not necessarily reflect clinical reality. Two recent systematic reviews that did include DAs found through the web—as well as academically developed DAs—found that the quality of DAs for breast and prostate cancer is relatively low.12 13 The authors conclude that there is a lot of variation between individual DAs, and assessment of DAs in other cancer domains is necessary to have a more accurate reflection of what is happening in clinical practice. Especially since many CRC patients have low literacy skills,14 it seems crucial to assess the information and communication in DAs aimed at supporting patients with CRC with shared decision making. The focus of the current review is on stage I–III CRC DAs, where curative treatment is the main goal (and are therefore distinctly different from stage IV DAs). Currently, we are aware of only one systematic review that focusses on CRC DAs for treatment. However, this small review15 only included three academically developed DAs. Another systematic review assessed the usefulness of metastatic CRC nomograms16 (N=14). Both reviews conclude that quality of DAs for CRC is generally low and few patient DAs for CRC have been developed. The aims of this systematic review are therefore to (1) create a larger corpus of all existing treatment DAs for stage I–III CRC found both through scientific literature as online searches, (2) to get a deeper understanding of the general quality of CRC DAs and (3) to assess the communicative quality of such DAs.
Materials and methods
Patient and public involvement statement
With this study, we aimed to create a more accurate depiction of clinical practice for patients by not only including academically validated DAs, but also DAs that patients could find online. No patients were involved in the design or production, or in any other aspect of this systematic review.
Search strategy
To identify DAs for patients with stage I–III CRC, a systematic academic and online literature search was performed in concordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.17 The MEDLINE, Embase, CINAHL, Cochrane Library and PsycINFO databases were searched from 2006 (as this is the launch date of the International Patient Decision Aids Standards, one of our assessment instruments) to February 2019. The search strategy (see online supplemental appendix 1) was adapted from earlier systematic reviews for prostate12 and breast13 cancer DAs and checked by a CRC clinician (IdH). References and author names of the studies found were checked for additional eligible DAs. The Ottawa Decision Aid Library and The International Database for Support in Medical Choices (Med-Decs) were also consulted. Languages included were Dutch, German and English.
Supplemental material
To ensure that all DAs accessible to patients were incorporated into our analysis, we also performed a Google and Bing search in Dutch, German and English (search date: 16 April 2019). Search terms were: “colon/rectal/colorectal cancer” (DU: “(dikke)darm/anus/endeldarm-kanker”; GER: “Darm-/Mastdarm-krebs)’ + “decision aid” (DU: “keuzehulp”, GER: “Entscheidungshilfe”). We searched the first 100 hits.
Selection criteria
For the academic literature search, studies that were published in peer-reviewed scientific journals between 2006 and 2019 and that were written in English, Dutch or German could be included. Papers that described randomised controlled trials, experiments, the development or evaluation of DAs could be selected.
For both the academic articles and the online search, only tools aimed at supporting the decision-making process of colon, rectal and patients with CRC stage I–III were eligible for selection. Tools targeted only towards metastatic colon, rectal or CRC patients were excluded from the analysis, as were tools aimed at screening decisions for patients with CRC. These tools are focused on inherently different decisions (eg, ‘should I get a screening test’ or ‘which treatments can help with quality of life for my final stages of life’) and therefore require different communication strategies. Appropriate formats for DAs were considered paper-based DAs (booklets or pamphlets), web-based DAs (websites), computer-based DAs (computer programs) and videos. Additionally, DAs had to be freely available, refer to at least two treatments, and written in English, German or Dutch. Nomograms as well as focus groups and question prompt sheets were not included as they cannot be analysed with the assessment instruments we use.
Data extraction
Two reviewers (SH and FC) screened all the retrieved articles and selected eligible articles based on titles and abstracts. After initial screening, disagreements were resolved via discussions with a third reviewer (RV). Full articles were independently assessed using a predefined selection checklist (see online supplemental appendix 2) by two reviewers (SH and FC), and final decisions about inclusion were made jointly with a third reviewer (RV). Inter-rater agreement was substantial between the reviewers (κ=.79). The data extraction forms (online supplemental appendix 3) were filled out independently by two reviewers (SH and FC). Both the selection and the data extraction forms were based on earlier systematic reviews12 13 to ensure consistency between outcomes.
Assessment instruments
Two instruments were used to assess the quality of the communication within DAs: the International Patient Decision Aid Standards (IPDAS) and Communicative Aspects (CA) checklist. Five teams of coders, containing two reviewers each, were responsible for the assessment (see online supplemental appendix 4 for a full overview). This way, each DA was reviewed by two coders using both the IPDAS and CA checklist. To calculate inter-rater agreement between team members, we used the Kappa statistic (κ). Although there has been some debate about the assumptions underlying the kappa statistic,18 19 we decided to keep the measure as they are well understood and frequently used to compute inter-rater agreement. We have, however, also provided the agreement matrices so other agreement indices may be calculated (see online supplemental appendix 5).
The IPDAS instrument consists of 36 items (see table 1). It was developed by a group of clinical researchers, practitioners and stakeholders20 to ensure that DAs adhere to certain quality standards21 and has been validated (for more detailed information on the validation process see: Elwyn et al and the associated website http://ipdas.ohri.ca).20–22 The instrument is divided into nine key components: information, outcome probabilities, clarifying values, decision guidance, developmental process, using evidence, disclosure and transparency, plain language and evaluation. As the validity of DAs was not assessed academically for all DAs, the evaluation dimension was excluded from analysis. Items could have the values ‘yes’ (1) or ‘no’ (0). Final scores were converted to percentages of the total number of items.
Results from the International Patient Decision Aids Standards (IPDAS) of colorectal cancer patient decision aids
The CA checklist was developed and validated by an interdisciplinary team of communication researchers and medical psychologists12 13 to create an in-depth quality assessment of the communicative quality within DAs (see table 2). With ‘communicative quality’ they mean the assessment of whether or not there is evidence that ‘the communicative process in which shared decision-making occurs (Vromans et al, p.2)’ is sufficient. The checklist consists of 76 items and has questions relating to seven main domains (1) information presentation, (2) information control, (3) personalised information, (4) interaction, (5) accessibility, (6) suitability and (7) source of information. Valid responses are ‘yes’ (1) or ‘no’ (0), and final scores are computed in percentages of the total number of items. The total number of items for paper-based DAs was 70 (as not all items were applicable) and 76 for web-based/video-based DAs.
Results from the Communicative Aspects checklist of colorectal cancer patient decision aids (DAs)
Note that for both assessment instruments (IPDAS and CA) a higher score does not necessarily mean that the quality of such a DA is higher, it merely indicates that more aspects have been taken into account.
Results
Study selection
Initially, 5645 unique studies were found through the systematic literature search (see figure 1 for a flow chart of the complete study selection). After eligibility checks through abstract and full-text screening, 121 studies were selected. In the end, 18 DAs were identified through the academic literature search (n=1) and online sources (n=17). These numbers compare to earlier systematic reviews using the same assessment instruments.12 13
Flow chart of the study selection for academic literature and online search.
Additionally, we updated the search between submission and revision. On 16 February 2021, we ran the Google Search again in German, Dutch and English. No new DAs were found in this search in Dutch and English. We did find some updated versions of DAs in German (namely: DA2, DA4 and DA5) but after careful comparison we concluded that no changes were made that impacted the scores of the original DAs so we decided not to replace them. We also ran our academic search in PubMed once more for the period of April 2019–February 2021. This search identified 378 articles, but after title and abstract screening, none of the articles were selected for inclusion. Reasons for exclusion were: ‘no DA discussed’, ‘not a treatment DA for CRC stage I–III’ (eg, a DA about screening decisions or metastatic cancer) and ‘not a DA but a nomogram’.
Table 3 shows a detailed description of the DA characteristics including titles of the DAs, developing organisations, country of origin (AUS/USA/IE/CAN=10, GER=6, NL=2), target audiences, treatments discussed, year of publication, DA format (web=1, video=1, paper/PDF=16) and length of the DA (min=2 pages, max=127 pages).
Characteristics of included decision aids
The IPDAS results
Inter-rater agreements (κ) between teams ranged from fair to substantial agreement (κ=.32 to κ=.60) for IPDAS. As is visible from table 4, IPDAS scores for individual DAs ranged from 28% to 78% (mean=48%, SD=14.15%, first quartile=38%, third quartile=58%, median=47%). The best performing DA was DA1, which also was the only DA with an associated research paper. Three DAs (DA6, DA11 and DA15) only scored 28%, which means that they met 10 of the 36 IPDAS items. In figure 2, a visualisation of the IPDAS results is shown.
Individual IPDAS item scores per DA
Violin plot of the IPDAS results. IPDAS, International Patient Decision Aid Standards.
All IPDAS items can be found in table 4. In total, there are 18 DAs and 36 IPDAS items. IPDAS scores are the sum of all 36 items per individual DA. The %IPDAS score is the percentage of IPDAS items met per individual DA (max=100).
Information
All DAs (N=18) described the health condition, the index decision and the options available for that decision. However, less than half (n=8, 44%) described the natural course of the disease if no action was taken. Positive features of specific treatment(s) at hand were shown by 8 DAs (44%), whereas 13 DAs (72%) offered negative features of treatment(s). Only three DAs (DA1, DA8 and DA9) (17%) allowed for a fair comparison between treatment options, and six DAs (33%) explained the different treatments with equal detail.
Outcome probabilities
Almost all DAs (n=16, 89%) described the likely consequences of the decisions (the outcome probabilities). More than half (n=11, 61%) explained uncertainty around probabilities. Additionally, 50% of DAs provided the reference class, used multiple methods to view the probabilities and specified the time period over which the outcome probabilities applied. Eight DAs (44%) discussed event rates, and only five DAs (28%) provided the outcomes probabilities in a balanced way and used the same denominator for the outcome probabilities.
Clarifying values
About 70% of DAs clarified to patients what it is like to experience the physical (72%) and psychological (67%) consequences of certain treatments. The social consequences were explained in 56% of DAs, and even fewer DAs (44%) expressed that patients had to think about what positive or negative features of the decisions matters most to them.
Decision guidance
Decision guidance was provided by leading patients in a step-by-step way through the decision (67% of DAs) and/or providing a list of questions to ask their clinician (61% of DAs).
Developmental process
Although 61% of DAs reported that the DA was reviewed by clinicians, only 33% mentioned the review involvement of patients in this process. Only three DAs (17%) mentioned that patients were asked about their needs for the DA, and one DA (DA10) mentioned that clinicians were asked about their needs for the DA. Similarly, only one DA mentioned that it was tested with patients (DA1), and none of the DAs mentioned that they were tested with doctors.
Using evidence
Half of the DAs provided a publication rate and an update policy for the DA. Only 28% of DAs (n=5) provided the reader with citations of the evidence used in the DA. Less than 20% of DAs reported about the quality of the evidence used (17%) and how the evidence was selected (11%).
Disclosure and transparency
More than 80% of DAs (83%) provided information about the authors and developers. About 70% (67%) also provided information about the funding related to the DA.
Plain language
Only 17% of DAs (DA1, DA9 and DA10) reported reading levels related to plain language.
The CA results
Inter-rater agreement (κ) for the CA checklist ranged from fair to substantial (κ=.38 to κ=.79). Results for the CA checklist ranged from 28% to 58% (mean=41%, SD=6.2%, first quartile=38%, third quartile=43%, median=41%). The DA that scores highest on the CA scale was DA9, while DA6 was the lowest on this scale. Table 2 shows an overview of the CA results, in figure 3, these results are visualised.
Violin plot of the CA results. CA, communicative aspects.
Information presentation
All DAs provided probabilistic information. As for the methods used to express them, all DAs reported verbal statistics (eg, ‘It is likely that you experience nausea’ or ‘most people have side effects’). Of the DAs that also included numerical probabilities (n=15, 83%), most reported natural frequencies (72%) (eg, ‘1 in 10 people …’). About one-third reported percentages (33%) (eg, ‘70% of the population …’). Absolute risks (eg, ‘The chance of recurrence of 60%, with chemotherapy this is 40%’) were given in 44% of DAs. Relative risks and relative risk reductions (eg, ‘compared with chemo, it is five times as likely to …’) were given in 17% of DAs. Five DAs provided the information visually (6% used a pie chart, 11% had a bar chart and 17% showed icon arrays).
Most DAs (n=16) provided information about the uncertainty around the information. All of the DAs that communicated uncertainty did this verbally, and 6 DAs also showed a numerical range (eg, ‘1 or 2 out of 10’ or ‘10%–20% of people’).
All but one DA (DA8) provided disease related information (eg, explain what (colorectal) cancer is), and 71% of DAs also included visuals to do so. Additionally, all but one DA (DA6) included information about the procedure of treatments discussed, and 59% also used visuals to explain this. There were two non-paper based DAs (DA9 and DA10), of which 1 offered audio and audiovisual stimuli to explain disease related information and procedures. There was also one paper-based DA that offered this (by providing web links) (DA12).
Almost all DAs (94%) used consistent fonts throughout the DA. Half of the DAs used unbiased language, and about two-fifth of DAs (44%) used roughly the same amount of text for each treatment option. Seventy-eight percent of DAs kept the order of positive and negative features of the treatments consistent. Only one DA that mentioned positive features of treatments (n=9) showed these with equal detail (DA8). Similarly, for the DAs that mentioned negative features of treatment options (n=16), the same DA (DA8) discussed them equally. Overall, 78% of DAs that mentioned both negative and positive options (n=9) kept the order in which they discussed these consistent.
Information control
Most DAs provided access to external sources (n=16). Also, most DAs (n=16) allowed patients to search for key words (as CTRL+F is always an option in PDF). Two-thirds of DAs (67%) provided patients with a step-by-step way through the DA and gave patients the opportunity to read more about specific topics. Less than half of the DAs (n=7) made it easy to return to previous sections of the DA (eg, by providing clickable links to earlier content or providing a contents ruler on each page). Three DAs provided access to internal sources (eg, ‘read/learn more’ sections) and two DAs (DA9 and DA14) provided patients with the option to only receive information that they would want to have (eg, by making it easy to skip sections).
Personalised information
In general, DAs contained few options to personalise information. Only five DAs (28%) were tailored towards specific disease factors and four DAs (22%) had tailored information for specific stages of the disease or the type of treatment(s) patients were eligible for. There was one DA (DA7) that made it possible to tailor the content and one DA (DA12) provided the option to change the mode of presentation (eg, by providing the same content in audio, video and text).
Interaction
More than half of the DAs (61%) mentioned that patients needed to assess their own personal values and preferences for the different treatment options. Of the DAs that offered this assistance (n=11), almost all did so in a passive way by either recommending patients to think about their personal preferences (n=10, 91%) and/or by asking patients for their preferences (64%, n=7). There were also 2 DAs that provided active interactions by giving weighting exercises (DA1 and DA9) and sliders (DA9).
There were four DAs (22%) that allowed for the comparison of negative and positive treatment options in an active way by verbally comparing the options (n=4) or providing a table with negative and positive features (n=3, 75%).
Four DAs (22%) showed the progress of the DA, whereas one DA (6%) provided a summary of the values and preferences of patients (DA9).
All the paper-based DAs (n=16) could be printed as one document, half of the DAs provided space for note taking and two DAs (11%) provided the patient with a short knowledge quiz (DA1 and DA9).
Accessibility
No DA required a login code and all DAs could be used on multiple devices. Almost all DAs were freely available on the web, not purely computer based and did not require internet access or staff assistance (n=17, 94%). Most DAs reported the last update (89%). Finally, 83% (n=15) of DAs were self-administered. However, only 33% of DAs (n=6) reported the update frequency.
Suitability
Although 72% of DAs (n=13) had a conversational style, up to 44% (n=8) contained irrelevant illustrations (eg, showing random people without providing any context). Also, almost all DAs were lengthy (lengthy >10 pages/5 min; n=15; min: 2; max; 127; M=58 pages), the video DA was 13:09 min (DA10)).
Source of information
There was one DA that reported on which dataset(s) the probabilistic information was based (DA10), the rest of the DAs did not report this. The most reported statistic was treatment side effects (n=13, 72%), followed by quality of life information (n=8, 44%), survival rate (n=6, 33%), mortality rate (n=5, 28%) and incidence (n=4, 22%). None of the DAs mentioned at what scale patient data have been collected, the number of patients the data are based on or the patient characteristics of the evidence used. Only two DAs (11%) mention the time period of the data collection.
Discussion
Our systematic review of 18 patient CRC DAs shows that the communicative quality of these DAs varies substantially between individual DAs.
Our results are in line with previous systematic reviews on CRC DAs in general15 and CRC Decision Support Systems for stage IV16 as both conclude that evidence for the quality of CRC DAs is too limited to recommend their use in clinical practice today. Additionally, conclusions can be drawn for the quality of communication in DAs between prostate,12 breast13 and CRC, as all reviews indicate that there are substantial differences in the communicative quality between individual DAs and overall quality seems to be low.
Strengths of this systematic review include the wide scope of our search, but also the in-depth analysis on the kind of information given in DAs for CRC. Our analysis showed that in most CRC DAs, probabilities are only communicated verbally. This is problematic, as research shows that people have a hard time interpreting verbally communicated statistical information23–26 such as ‘there is a big chance of …’. Additionally, information seems to be generic and lengthy in CRC DAs, whereas providing patients with personalised health information is recommended27 as this reduces the information overload patients may experience.28 Especially since many CRC patients have low health literacy skills,14 it seems crucial that information is (also) visualised29 30 and communicated in plain language.27 However, our analysis shows that this is often not the case. Finally, as in previous systematic reviews on treatment DAs,28 31 we found that many do not provide citations for the evidence used and they often seem to rely on anecdotal evidence instead.
We conjecture that many of these issues can be addressed using Natural Language Generation,32 an AI technique which automatically converts data into fluent and coherent text (possibly combined with automatically generated pictures), tailored to individual readers. A recent example harnessing these techniques for personalised DAs is a prototype decision support tool that generates personalised probabilities for effects on quality of life after chemotehrapy.33 In short, the support tool relies on the PROFILES34 registry data set, consisting of over 21 000 patients with cancer within the Netherlands Cancer Registry. With latent class analysis,35 the tool can predict which outcome scenario is most applicable for a new patient based on individual prognosis data and the PROFILES data set. This way, patients can view symptom-related quality of life outcomes such as the probability of becoming nauseous, but also social or financial implications of chemotherapy. We are currently evaluating the tool with different patients to see how we can communicate the different outcomes in a personal and accessible way.
There are also several limitations to this study. It should be noted that our review did not take measures of the effectiveness of DAs, such as decisional conflict or participation in shared decision-making, into account as this was not within the aims of our study. It should also be noted that although IPDAs and CA can be used to guide the design process of DA developers, using these tools does not ensure (communicative) quality. We, therefore, stress that DAs should also always be evaluated with clinical experts and patients.36 Finally, as we included several countries within our review, results appear to apply to all different countries. However, it seems to be the case that plain language use was harder to establish for the German DAs which might be because of the formal sentence structures in German. Additionally, it seemed that German patients were less encouraged to participate in shared decision making (‘listen closely to your doctor’) then, for example, American patients (‘make decisions you want to make’). Although it has been demonstrated that culture might impact the effectiveness of health communication between doctors and patients of different cultural background,37 38 studying cultural differences between (European) countries remains challenging as theories and methods for assessing differences vary between countries.39 Future reviews could look into systematic differences between DAs from different countries more to see if shared decision making is a globally agreed on goal.
Conclusion
This review is—to the best of our knowledge—the first to perform a large-scale analysis of the quality of communication in treatment CRC patient DAs. The findings highlight the variety of communicative quality in DAs and the lack of support that many DAs are able to provide to both patients and clinicians in shared decision making in a clinical setting. It calls for personalising information in CRC treatment DAs in order to facilitate patient participation in shared decision making. To ensure this, both the IPDAS instrument and CA checklist can be useful tools to guide DA developers in such a way that they are made aware of certain aspects and can take them into account. Future research should focus on evaluation of such personalised tools to test their usefulness in the clinical practice.
Data availability statement
Data are available on reasonable request. All data relevant to the study are included in the article or uploaded as online supplemental information. All reviewed materials were made freely available on the internet by their developers. A full list of the DAs and their titles can be found in table 3. Individual assessments of each reviewer are stored in a secure data repository at Tilburg University Dataverse and is available on request by contacting the main author.
Acknowledgments
For the overall project ‘Data-driven Shared Decision Making for Cancer Patients’, we received support from The Netherlands Organisation for Scientific Research (NWO Data2Person 628.011.030), for which we are grateful. We would also like to thank all authors and developers of the DAs involved for making their materials accessible. Also, many thanks to Marie Barking, our German colleague who helped with the assessment of the German DAs.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors CRediT author statement. SH: conceptualisation, validation, formal analysis, investigation, data curation, writing-original draft, writing-review and editing, visualisation, project administration. RV: conceptualisation, methodology, validation, formal analysis, investigation, resources, writing-review and editing, visualisation. FC: conceptualisation, formal analysis, investigation, writing-review and editing. XV: conceptualisation, formal analysis, writing-review and editing. IdH: conceptualisation, validation, resources, writing-review and editing. EK: conceptualisation, methodology, validation, formal analysis, writing-review and editing, supervision, funding acquisition.
Funding This systematic review is part of the Netherlands Organisation for Scientific Research (NWO Data2Person 628.011.030) project. IdH received unconditional research grants from QP&S/RanD, ROCHE Pharmaceutical and KWF for unrelated research.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.