Objective Health technology reassessment (HTR) is a field focused on managing a technology throughout its life cycle for optimal use. The process results in one of four possible recommendations: increase use, decrease use, no change or complete withdrawal of the technology. However, implementation of these recommendations has been challenging. This paper explores knowledge translation (KT) theories, models and frameworks (TMFs) and their suitability for implementation of HTR recommendations.
Design Cross-sectional survey.
Participants Purposeful sampling of international KT and HTR experts was administered between January and March 2019.
Methods Sixteen full-spectrum KT TMFs were rated by the experts as ‘yes’, ‘partially yes’ or ‘no’ on six criteria: familiarity, logical consistency/plausibility, degree of specificity, accessibility, ease of use and HTR suitability. Consensus was determined as a rating of ≥70% responding ‘yes’. Descriptive statistics and manifest content analysis were conducted on open-ended comments.
Results Eleven HTR and 11 KT experts from Canada, USA, UK, Australia, Germany, Spain, Italy and Sweden participated. Of the 16 KT TMFs, none received ≥70% rating. When ratings of ‘yes’ and ‘partially yes’ were combined, the Consolidated Framework for Implementation Research was considered the most suitable KT TMF by both KT and HTR experts (86%). One additional KT TMF was selected by KT experts: Knowledge to Action framework. HTR experts selected two additional KT TMFs: Co-KT framework and Plan-Do-Study-Act cycle. Experts identified three key characteristics of a KT TMF that may be important to consider: practicality, guidance on implementation and KT TMF adaptability.
Conclusions Despite not reaching an overall ≥70% rating on any of the KT TMFs, experts identified four KT TMFs suitable for HTR. Users may apply these KT TMFs in the implementation of HTR recommendations. In addition, KT TMF characteristics relevant to the field of HTR need to be explored further.
- public health
- health services administration & management
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This was the first study to solicit the perspectives of international health technology reassessment (HTR) and knowledge translation (KT) international experts on the suitability of KT theories, models and frameworks (TMFs) for HTR.
Purposeful and snowball sampling was employed to obtain HTR and KT experts from different jurisdictions with a depth and breadth of knowledge in both KT and HTR to ensure a representative sample.
Through a survey, experts were asked to rate each KT TMF as ‘yes’, ‘partially yes’ or ‘no’ on six criteria to select potential KT TMFs for HTR: familiarity, logical consistency/plausibility, degree of specificity, accessibility, ease of use and HTR suitability.
Only full-spectrum KT TMFs (KT phases: planning/design, implementation, evaluation and sustainability/scalability) were included as these phases are critical to the KT process and necessary for the HTR process.
The sample size of HTR and KT experts was small, which may have reduced the ability to generate consensus (≥70% experts selected ‘yes’) on a suitable KT TMF for HTR.
Health technology reassessment (HTR) is the systematic process of evaluating technologies that are currently in the health system to ensure that they are being used optimally.1 Recommendations from the HTR process can result in increase in use, decrease in use, no change or complete withdrawal of the technology.2 However, implementation of these recommendations has been challenging.2 It has been argued that the field of knowledge translation (KT) could play a role in the implementation process for HTR recommendations.3 KT has been described as ‘a dynamic and iterative process that includes the synthesis, dissemination, exchange and ethically-sound application of knowledge to improve the health of [populations], provide more effective health services and products, and strengthen the healthcare system’.4 In essence, KT is the application of putting knowledge into practice and policy. KT approaches could be used in the HTR process to bridge the gap between the generation of recommendations regarding optimal technology use and their implementation in practice.3 Thus, KT can be seen as complementary to the HTR process, but there has been a paucity of research in this area.3 Moreover, there is a gap in our understanding of which KT theories, models or frameworks (KT TMFs hereafter) would be best suited for the translation of HTR recommendations.3
In the literature, two narrative reviews and two scoping reviews have reported from 41 to 159 KT TMFs depending on how they are identified and considered.5–8 KT TMFs have been used in different contexts, settings and populations.5–8 Moreover, there has been some use of the KT interventions, strategies and TMFs to decrease or remove low value care.9 10 These KT TMFs have been used to help identify determinants, barriers and enablers to behaviour change related to HTR.11 12 However, the use of these KT TMFs has not been applied consistently to the development of KT interventions or the field of HTR.3 13 There are also no recommendations about which KT TMFs could be used for HTR. Through an international survey of KT and HTR experts, this study aims to provide an understanding of which KT TMFs could be appropriate for the HTR process and implementation of its recommendations.
This study used three approaches to the selection of KT TMFs for HTR: identification of suitable KT TMFs, consensus on the list of KT TMFs through a modified Delphi process, and selection of potentially suitable KT TMFs through a survey of international KT and HTR experts.
Identification of suitable KT TMFs
Only full-spectrum KT TMFs were included. ‘Full-spectrum’ includes all four KT phases: planning/design, implementation, evaluation and sustainability/scalability.8 These four KT phases are critical to the KT process and are thought to be necessary for the HTR process and the implementation of its recommendations.3 A recent scoping review provided a preliminary list of 26 full-spectrum KT TMFs within cancer and chronic disease management contexts.8 A recent update of this scoping review conducted by the authors resulted in 36 full-spectrum KT TMFs identified.14 Eighteen were process models, eight were classic theories, three were determinant frameworks, three were evaluation frameworks and four fit more than one approach category.14 This list of 36 full-spectrum KT TMFs provided the initial list of KT TMFs to assess for use when implementing HTR recommendations.
Consensus on the list of KT TMFs using a modified Delphi process
To ensure that the list of 36 full-spectrum KT TMFs was adequate and concise, a convenience sample consisting of the authors of this study reviewed this initial list to determine if any KT TMFs had been missed or could be eliminated based on HTR suitability. This sample was considered suitable as the authors had clinical training combined with expertise in KT or HTR and/or were experts at the doctorate level in these fields. A three-round modified Delphi process was undertaken.15–17 The Delphi process is iterative and used to determine expert group consensus where there is a lack of evidence and expert opinion is important.18 The first and second rounds involved independent review of each KT TMF to determine which would be suitable for HTR. Each author rated the KT TMF as ‘yes’, ‘potentially yes’ or ‘no’ for HTR suitability. Consensus to keep the KT TMF was defined as 100% of the authors rating the KT TMF as ‘yes’ and/or ‘potentially yes’. Consensus to eliminate the KT TMF was defined as 100% of the authors rating the KT TMF as ‘no’. Any KT TMFs that did not reach consensus were discussed in subsequent rounds. The third round entailed a 2-hour face-to-face meeting held in October 2018. Prior to the discussion at this meeting, the authors agreed on ground rules, principles and criteria for selection of KT TMFs for HTR suitability (box 1). The authors deliberated on the remaining KT TMFs until consensus was reached. Verbal consent from the participants was obtained prior and the meeting was recorded.
List of criteria developed by authors for round 3 of modified Delphi process
The KT TMF must have face validity (KT TMFs that are common and well known should be included).
The KT TMF must be active KT TMF (passive KT TMFs were excluded).
The KT TMF must be feasible to apply to take something out of practice.
The KT TMF was pragmatic (theoretical KT TMFs were excluded).
The KT TMF must be specific (vague or those that were not prescriptive were excluded).
The KT TMF could build on other KT TMFs but needed to be generic rather than for a specific context.
The KT TMF is easily understood and practical.
Any KT TMF that the committee was undecided on.
International expert survey
Selection of experts to review KT TMFs for HTR
HTR and KT experts were selected through purposive and snowball sampling. Names were initially derived through the KT Canada website, Health Technology Assessment International Disinvestment Interest Group, authors of relevant publications and in consultation with other experts. A list of HTR and KT international experts was generated by country, including Canada, USA, UK, Australia and European countries (Germany, Italy, Sweden, Spain). Experts were contacted via email to participate in the study. They were sent an email, invitation letter and information sheet. If they agreed to participate, they were sent a consent form, a survey with the list of KT TMFs identified by the modified Delphi process to rate (online supplemental file 1) and a recent article on the topic as background information.3 If they were unable to participate, the next expert name on the list was contacted. This was done to ensure that there were at least two HTR and two KT experts from each of Canada, USA, UK, Australia (n=16) and four HTR and four KT experts from other European countries combined (n=8), for a target sample size of 24. Experts contacted could also suggest additional names of experts to be surveyed through snowball sampling. These names were added to the list of experts and contacted, if required, to reach a predefined number of participants. Representativeness was assessed by ensuring that experts came from different jurisdictions with a depth and breadth of knowledge in both KT and HTR.
The Enhancing the Quality and Transparency of Health Research good practice in the conduct and reporting of survey research guidelines was followed for the development of the survey.19 The survey included the list of KT TMFs and a description of each KT TMF, followed by a link to the paper that described the KT TMF, if one was available. Specific criteria used previously to select KT TMFs were used to rate each KT TMF.20 These included familiarity, logical consistency/plausibility, degree of specificity, accessibility, ease of use and HTR suitability. Each criterion was operationally defined and reviewed by FC and HMH (online supplemental file 2). There was also a section for open-ended comments. The survey was developed in Excel and pilot-tested by four participants to ensure flow and functionality.
The survey was administered via email to the experts starting in January 2019. Based on the criteria, each KT TMF was rated by each expert as ‘yes’, ‘partially yes’ or ‘no’ and additional comments could be provided. Experts were also asked to suggest additional full-spectrum KT TMFs that could be suitable for HTR and recommend other experts that could be contacted for the study. Consensus was determined as ≥70% of experts selecting ‘yes’ for the particular KT TMF. The principles and criteria described in box 1 were also shared with the international experts for information purposes. Experts were asked to return the survey within 2 weeks. Two additional reminders were sent. If surveys were not returned, then another expert on the list was contacted to participate. The survey was sent out to experts until 31 March 2019 to ensure that at least two HTR and two KT experts had agreed to complete the survey from the identified countries.
Modified Delphi process
After rounds 1 and 2 of the modified Delphi process, data were analysed descriptively by tabulating the ‘yes’, ‘potentially yes’ and ‘no’ responses for HTR suitability for each KT TMF reviewed by the authors.
Survey data were analysed descriptively by tabulating the ‘yes’, ‘partially yes’ and ‘no’ responses for HTR suitability for each KT TMF and by HTR and KT expert subgroups. KT TMF familiarity and missing data were also descriptively summarised.
Data from the open-ended comments section of the survey provided by the HTR and KT experts were analysed using content analysis.21 As these data were limited in volume, content analysis was undertaken to provide a starting point in determining preliminary factors that may be important to consider for a KT TMF for HTR.
Initially, all comments from each expert were entered into Excel and categorised by KT TMF. These were read and reread to get familiarised with the data. Next, for each KT TMF, each comment was organised by response to HTR suitability as ‘yes’, ‘partially yes’, ‘no’ and unfamiliar with the KT TMF. This categorisation provided an understanding of what comments may or may not be important to consider for HTR suitability. Open coding and constant comparison were applied inductively to all the comments. A preliminary list of codes, subcodes and operational definitions was developed manually through independent review of the comments from three KT TMFs (Consolidated Framework for Implementation Research (CFIR), Stages of Research Evaluation, and Knowledge to Action (KTA) framework) by RE and HMH. A final taxonomy consisting of codes, subcodes and operational definitions with exemplar quotes was applied manually to the comments for the remaining KT TMFs by RE (table 1). Manifest content analysis, defined as the development of categories, as opposed to latent content analysis (defined as the development of themes), was determined to be best suited given the nature of the open-ended comments.21 Categories were created, grouping codes under higher-order headings and formulating a general description of these categories. In addition, the frequency of comments for each code in each category was also tabulated by HTR and KT experts to determine the top categories/codes. The most prominent codes and interpretation of the data were determined through frequency counts, discussion and consensus between FC and HMH.
Patient and public involvement
Patients or the public were not involved in the design, or conduct, or reporting or dissemination plans of our research.
Modified Delphi process
The results of the modified Delphi process are presented in table 2. The third round resulted in the selection of 16 full-spectrum KT TMFs. There were 12 process models, 2 frameworks, 1 classic theory and 1 KT TMF that fit two categories (model and framework). Twenty KT TMFs were excluded. Fourteen were too vague and not descriptive enough, two were considered ‘passive’ and not ‘active’ KT TMFs to make change happen, two were not pragmatic, and two were too specific to a given context (ie, guideline adaptation and disability research) (online supplemental file 3).
International expert survey
Forty-eight KT experts and 31 HTR experts were invited to participate via email. A total of 22 experts (11 KT and 11 HTR) completed the survey. Experts were from Canada (n=4), USA (n=5), UK (n=3), Australia (n=4), Germany, (n=2), Spain (n=1), Italy (n=1) and Sweden (n=2). Of the experts, 59% were women and all had graduate-level education (master’s or PhD).
Overall, of the 16 KT TMFs, none received a ‘yes’ rating for HTR suitability by ≥70% of the experts. The top three most highly rated KT TMFs were CFIR,22 KTA23 and the Plan-Do-Study-Act (PDSA) cycle.24 Of the experts, 38% rated CFIR as ‘yes’, followed by 27% each for the KTA framework and the PDSA cycle.24 The least rated KT TMFs by the experts were the KT framework for Agency for Healthcare Research and Quality Patient Safety Portfolio and Grantees,25 the Stages of Research Evaluation,26 and the Staged Model of Innovation Development and Diffusion of Health Promotion Programs,27 which all received 0% ratings for ‘yes’ from the experts (figure 1). Combination of the ‘yes’ or ‘partially yes’ ratings found that 86% (19 of 22) of the experts selected CFIR as the top rated KT TMF for HTR suitability.22
Stratified analysis by KT and HTR expertise
KT experts favoured KTA (82%, 9 of 11) as another KT TMF that would be suitable for HTR,23 in addition to CFIR (91%, 10 of 11). HTR experts favoured the Co-KT framework (72%, 8 of 11)28 and the PDSA cycle (72%, 8 of 11),24 in addition to CFIR (82%, 9 of 11).
Of the comments, 49% provided by both KT and HTR experts were related to the TMF characteristics category, followed by the TMF attributes category (19%). Implementation and user categories both had 13% each (figure 2).
Overall, the top code was ‘pragmatic’ under the TMF characteristics category (14%), defined as the KT TMF not being theoretical but practical and application of the TMF outside of research or academic settings. This was followed by implementation (13%), defined as the KT TMF providing operation detail on how to ‘do’ the implementation to achieve the HTR outputs. This included exploring determinants, their interrelationships and the development of interventions or strategies based on these determinants. The third top code was HTR suitability under the TMF attributes category (8%), defined as a ‘strong fit’ to HTR and its determinants. It also included the ability to adapt the KT TMF and tailor it to micro (individual), meso (organisational) and macro (policy) levels.3
More KT experts than HTR experts commented on pragmatic as an important characteristic of a KT TMF (56% vs 44%). There were both positive and negative comments related to pragmatic that would make a KT TMF suitable for HTR. For example, one KT expert who said ‘yes’ to HTR suitability for the PDSA cycle noted the following positive affect:
A basic, simple but still very useful approach. (009)
In contrast, in reference to the Stages of Research Evaluation, one HTR expert who said ‘no’ to HTR suitability stated the following negative affect:
This is also difficult to be implemented in reality as it is far from explaining the characteristics of the healthcare systems and professional interactions. (017)
More KT experts than HTR experts provided comments related to implementation (78% vs 22%). There were both positive and negative affects of comments related to implementation that would make KT TMF suitable for HTR. One KT expert who said ‘yes’ to HTR suitability for the Quality Implementation Framework stated the following positive affect:
I’m not familiar with specifics about this framework; it certainly covers the full-spectrum of considerations for implementing new interventions; could be adapted for de-adoption/implementation. (005)
On the contrary, another KT expert who said ‘no’ to HTR suitability with respect to the Diffusion of Innovations theory stated the following negative affect:
I think (as it is a general theory rather than an implementation framework/model) that it lacks sufficient guidance on how to implement/de-implement. (007)
More KT experts provided comments to HTR suitability than HTR experts (60% vs 40%). There were both positive and negative affects of comments related to HTR suitability for a KT TMF. One HTR expert who said partially ‘yes’ to HTR suitability for CFIR stated the following positive affect:
A lot of constructs have been included in CFIR, so in each case, it would probably require selection of the specific ones relevant for the HTR example. (021)
Another KT expert who said ‘no’ to HTR suitability for the CollaboraKTion Framework stated:
Depends on focus of work-this emphasizes need for community to decide on action whereas if you had a particular output in mind to implement/de-implement this might not be the best fit. (001)
However, HTR experts commented more on the ability to tailor the KT TMF to micro, meso and macro levels than KT experts (90% vs 10%).
The focus of this study was to determine KT TMFs that could be suitable for implementation of HTR recommendations. Three key findings emerged: (1) ≥70% consensus (rated as ‘yes’ by the experts) was not reached by the international KT and HTR experts on any of the current full-spectrum KT TMFs; however, when ratings of ‘yes’ and ‘partially yes’ were combined, CFIR was considered the most suitable KT TMF by both KT and HTR experts; (2) KT experts identified one additional KT TMF, the KTA framework, whereas HTR experts identified two additional KT TMFs, the Co-KT framework and the PDSA cycle, as potentially suitable for HTR; and (3) overall, experts commented on three key characteristics of a KT TMF that may be important to consider: practicality, guidance on how to implement and adaptability of the KT TMF to HTR.
This study used a modified Delphi process and survey to illicit input from study authors and international KT and HTR experts. Although experts may not have sufficient knowledge of all the KT TMFs, this was the first study that attempted to garner the opinions of experts in both fields. The field of KT and its application to HTR has been proposed as a mechanism to advance the implementation of HTR recommendations into practice.3 The selection of one determinant framework (CFIR) and three process models (KTA framework, Co-KT framework and the PDSA cycle) provides a starting point of potential KT TMFs that could be used with HTR. However, as ≥70% consensus was not reached by the experts, these findings need to be considered as preliminary.
The Delphi technique has been criticised for lack of guidelines on the determination of the size of the expert panel, lack of anonymity, what is meant by ‘expert’ opinion and determination on the level of consensus.29 The sample size of five may have been too small to select KT TMFs from a list of 36 full-spectrum KT TMFs. The 100% consensus level may have been too high. There may also have been pressures of conformity at the face-to-face meeting. However, the authors had a wide range of expertise in HTR, KT or both. The use of a facilitator and establishment of ground rules and principles upfront were important considerations to address pressures of conformity.
Although purposeful sampling was used for the survey, the sample size of international KT and HTR experts was small, which may have reduced the ability to generate consensus. However, considerable efforts were made to target experts with knowledge and practical experience in KT and/or HTR. Lastly, the selection of 70% consensus was arbitrary and determined a priori to survey administration. This level of agreement has been considered appropriate in previous Delphi studies,30 but there is no acceptable level of consensus.29
Implications of findings
Among the list of 16 full-spectrum KT TMFs identified through a modified Delphi process, the international experts were not able to select a current KT TMF for HTR. Lack of familiarity with the KT TMFs could be one reason. Specifically, experts were not familiar enough with 4 of the 16 KT TMFs to rate them for HTR suitability. Over the recent years, there has been a flurry of KT TMFs developed.8 This proliferation of KT TMFs makes it challenging for experts to keep abreast of them. Moreover, there has been criticism of the development of KT TMFs without adequate testing, validation and research.31 Experts within the KT field may lean towards those KT TMFs that they are most familiar with.8
Another reason experts were challenged to select a KT TMF may be the lack of understanding of the HTR process. KT experts in particular may have found it difficult to review the KT TMFs and then apply them to HTR, as they may not be familiar enough with the HTR process itself. HTR has also been confused with terms such as ‘disinvestment’ and ‘de-adoption’, which are considered outcomes of the HTR process rather than the process itself.2 In addition, the field of HTR is underdeveloped and concepts have yet to be agreed on.3 An information sheet and background paper with a description of the fields of KT and HTR was provided to the experts prior to the survey. However, these materials may not have been reviewed in advance or been a sufficient knowledge resource.
CFIR was the only KT TMF selected by both HTR and KT experts as a potential KT TMF that could be used for HTR. CFIR has been used widely and is a well-operationalised, multilevel implementation determinant framework derived from theory.32 33 The application of CFIR and its constructs may enable users to assess facilitators and barriers to the implementation of HTR recommendations, particularly when HTR recommendations result in decreased use or removal of the technology. The assessment of facilitators and barriers has been noted as an important step within the de-adoption process of low value care.3 34 However, future research with a focus on the application of CFIR to HTR projects is needed.
The KTA framework was primarily selected as suitable for HTR by KT experts. Its selection could be due to its widespread use in the KT field.35 In fact, one adaptation of the KTA framework has been the Synthesis Framework for Facilitating De-adoption.34 This framework has been proposed for potential use in HTR projects.3 However, it has yet to be applied in practice. Nonetheless, the KTA framework’s ability to be adaptable may be another factor in its selection primarily by KT experts.
The Co-KT framework28 and PDSA cycle24 were primarily selected for HTR suitability by HTR experts. Both are process models.14 The Co-KT framework is a linear process and may be considered simplistic to apply. The PDSA cycle has been used extensively in quality improvement as a model for change.36 It is a simple and pragmatic model to use and is adaptable within other models.37 However, it is not without its limitations.36 Subsequently, selection of these KT TMFs by HTR experts may be due to their ease of use.
Implications for future research
Although not the key focus of this study, three key characteristics—practicality, guidance on how to implement and adaptability of the KT TMF to HTR—were identified from the open-ended comments. These key characteristics and others may be important to further interrogate. Future research on identifying the key elements, attributes and constructs of KT TMFs for HTR through expert interviews is needed to better understand which would influence and demonstrate an important role for HTR.
Recently, there has also been a proliferation of disinvestment frameworks or frameworks to address overuse.13 38 39 Some are based on KT and implementation science principles.13 The focus of these frameworks has been on removing or reducing low value care from practice. The application of these frameworks is still in its infancy. Although the list of full-spectrum KT TMFs that were examined in this study did not consider these disinvestment frameworks, there may be merit in doing so.
This study provided insights into which KT TMFs may be suitable for HTR. Despite not attaining ≥70% rated as ‘yes’ on any of the KT TMFs through the survey, experts identified four KT TMFs that could potentially be used within the context of HTR (CFIR, KTA, Co-KT and PDSA). Familiarity, adaptability and ease of use may be some of the reasons that led to their selection. Moreover, characteristics of practicality, how to implement HTR recommendations and adaptability of the KT TMF to HTR need to be interrogated to determine if they are important in a KT TMF for HTR. The process of HTR could benefit from the field of KT and its application of KT TMFs in implementation of its recommendations. Future research on the application of KT TMFs to HTR projects will provide much needed guidance and advancement in this area.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics approval was obtained from the University of Calgary’s Conjoint Health Research Ethics Board (REB#17-0932).
We would like to thank all the individuals who participated in this study for their support and contributions to this work.
Contributors RE conducted the study, collected, analysed and interpreted the data. RE, FC and HMH drafted the manuscript. RE, FC, HMH, JH-L and DN contributed to study conception and design, planning and data interpretation, and were involved in revising the manuscript for important intellectual content. RE, FC, HMH, JH-L and DN read, provided edits and approved the final manuscript.
Funding RE is funded through an Alberta Innovates - Health Solutions Graduate Studentship Award.
Disclaimer The funding body did not participate in the design of the study and in the collection, analysis and interpretation of data or writing of the manuscript.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.