Article Text

Original research
The relationship between the living lab approach and successful implementation of healthcare innovations: an integrative review
  1. Nina Zipfel,
  2. Bedra Horreh,
  3. Carel T J Hulshof,
  4. Angela G E M de Boer,
  5. Sylvia J van der Burg-Vermeulen
  1. Department of Public and Occupational Health, Coronel Institute of Occupational Health, Amsterdam Public Health Research Institute, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands
  1. Correspondence to Dr Nina Zipfel; n.zipfel{at}amsterdamumc.nl

Abstract

Objectives The concept of living labs as a research method to enhance participation of end-users in the development and implementation process of an innovation, gained increasing attention over the past decade. A living lab can be characterised by five key components: user-centric, cocreation, real-life context, test innovation and open innovation. The purpose of this integrative literature review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations.

Methods An integrative literature review searching PubMed, EMBASE, PsycINFO and Cinahl databases between January 2000 and December 2019. Studies were included when a living lab approach was used to implement innovations in healthcare and implementation outcomes were reported. Included studies evaluated at least one of the following implementation outcomes: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration or sustainability. Quality was assessed based on a tool developed by Hawker et al.

Results Of the 1173 retrieved articles, 30 studies were included of which 11 of high quality. Most studies involved a combination of patients/public (N=23) and providers (N=17) as key stakeholders in the living lab approach. Living lab components were mostly applied in the development phase of innovations (N=21). The majority of studies reported on achievement of acceptability (N=22) and feasibility (N=17) in terms of implementation outcomes. A broader spectrum of implementation outcomes was only evaluated in one study. We found that in particular six success factors were mentioned for the added-value of using living lab components for healthcare innovations: leadership, involvement, timing, openness, organisational support and ownership.

Conclusions The living lab approach showed to contribute to successful implementation outcomes. This integrative review suggests that using a living lab approach fosters collaboration and participation in the development and implementation of new healthcare innovations.

PROSPERO registration number CRD42020166895.

  • quality in health care
  • public health
  • change management

Data availability statement

Data are available on reasonable request. All data generated or analysed during this study are included in this published article and its online supplemental information files.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • Summarises the literature on the relationship between the living lab approach and successful implementation of healthcare innovations based on the implementation outcomes suggested by Proctor et al.

  • Includes a broad search terms in order to understand what components of the living lab approach are currently applied.

  • Studies were included irrespective of study design (integrative review) and successful implementation was evaluated even when only one implementation outcome was reported.

  • As a shortcoming is the use of the Hawker et al quality appraisal tool as the tool itself does not suggest cut-off values for the overall quality assessment.

  • Most studies only evaluate acceptability and feasibility in terms of implementation and not the full range of suggested implementation outcomes by Proctor et al.

Background

The concept of living labs as a research method to enhance participation of end-users in the development and implementation process of an innovation gained increasing attention over the past decade.1 In Europe, the application of living labs in real-life settings and ‘real’ experimentation emerged around 2005. In line with strengthening democratic processes in the EU, policies strongly encourage collaborative approaches in order to create innovation and the involvement of stakeholders by including them into the design and implementation of different fields of research and development.2 At first, living labs mostly emerged from Information and communication technology and urban developments to test innovations in a real-world environment.3–5 In 2006, the European Network of Living Labs (ENoLL) was formed as an international collaboration platform. The living lab approach has, in the following years, also been adopted in healthcare settings with the introduction of several European living labs.6–10 Programmes, such as Horizon 2020, promote the use of the living lab approach, including its application in the field of healthcare and health promotion. In 2018, more than 440 living labs have been recognised in Europe. ENoLL identified 69 of the current living labs as being health-related concerning diverse topics such as ageing, healthy living and mobility, chronic diseases and technological innovation.11 Currently, no consistent or commonly accepted definition of living labs exists, but the following terms are considered key components of living labs: user-centric, cocreation, real-life context, test innovation and open innovation.1 3 12–15 The goal of living labs is to develop useful and usable products and/or services to create value.1 15 Also, in terms of the current discussion on political agendas concerning the involvement of the public in innovations, living labs offer the possibility to engage the public in the process of innovation development and implementation.16 For the purpose of this integrative review, a living lab is defined as a user-centric research methodology for developing, testing and implementing complex healthcare innovations in a real-life context. An example of a Dutch living lab is the eLabEL project which aimed to improve integrated digital support in primary care. Stakeholders consisting of patients, healthcare professionals, entrepreneurs and researchers collaborated during the selection, integration, implementation and evaluation of developed eHealth-tools in primary healthcare.17 In the living lab, stakeholders together identified needs and expectations of eHealth solutions followed by several sessions to integrate the chosen eHealth solutions.17 In this sense, complex innovations include ideas, practices or technologies that are new to the end-user and that require the involvement of multiple stakeholders to use and implement to achieve better quality of care.18

A recent literature review explored the concept of living labs to investigate population specific health-related problems and the application of five common elements of living labs, namely multimethod approach, user engagement, multiple stakeholders, real-life settings and cocreation.19 The authors found that all of the five key elements were used in most studies. Moreover, this review suggests that using a living lab approach helps to improve physical, social and cognitive health. However, a living lab approach does not exclusively concern the developmental process of products and/or services, but also ensures sustainable implementation.20

Implementation of research findings are essential to enhance timely adoption to improve quality of care.21 It is estimated that approximately two-thirds of efforts to implement change are not successful.22 Possible barriers include, for example, awareness, motivation to change, attitude and involvement on an individual professionals’ and patients’ level.23 Implementation concerns a set of purposeful processes and/or activities specifically developed to put an intervention or programme into practice.24 In order to assess successful implementation, measures need to be used that are distinct from those that assess effectiveness of an intervention. This distinction is crucial, as success or failure of innovation can be due to, for instance, an ineffective intervention or insufficient reach and/or incorrect use in practice. The incorrect use in practice and insufficient reach concerns assessment of successful implementation. Barriers to successful implementation include insufficient involvement and support, poor dissemination strategies and lack of leadership and willingness to change.25–27 Outcome measures for implementation have been proposed by Proctor et al and include the following: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration and sustainability.28 Successful implementation is proposed to be measured as an equation of the effectiveness of an intervention being implemented plus the described implementation factors.28 For the purpose of this integrative literature review, successful implementation will be measured as proposed by Proctor et al based on the implementation outcomes.28 The living lab approach may support successful implementation, as end-users are not only involved in the development but also testing of the innovative products. Therefore, the goal of this integrative literature review is to assess the literature on the relation between the living lab approach and successful implementation of innovations. By doing so, the aim is to (1) identify which key components of the living lab approach were used; (2) identify which implementation components were measured; and (3) determine what the relationship is between the living lab approach and successful implementation of innovations.

Methods

This integrative review aimed at including all available literature in the field to draw an understanding of the relation between the living lab approach and successful implementation of innovations. Methods of an integrative review allow for the inclusion of different study designs (qualitative and quantitative).29 Results of the review are reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (online supplemental file 1).30

Information sources and literature search

To identify relevant publications, a systematic search was performed in the bibliographic databases PubMed, Embase, Cinahl and PsycINFO from January 2000 to December 2019. Additionally, snowball strategies were used to screen reference lists of eligible papers. Search terms included free-text terms to capture the concept of “living lab” (eg, ‘co-creation’ or ‘co-design’) and “successful implementation” (eg, ‘fidelity’ or ‘implementation evaluation’). The concept of “successful implementation” entails the evaluation of at least one of the Proctor et al implementation outcomes. An information specialist was consulted in the development phase of the search strategy. The full search strategy tailored for all databases can be found in online supplemental file 2.

Study selection

The goal was to include studies that used a living lab approach in either of the following phases of an innovation: development, implementation or evaluation. Studies that report on a minimum of one implementation outcome were included in this study. For the purpose of this integrative review, implementation was defined as purposeful activities designed to put a programme or activity into practice.24 Studies evaluating or assessing at least one or more of the following implementation outcomes as proposed by Proctor et al were eligible for inclusion to evaluate successful implementation: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, sustainability.28 The implementation outcomes are described in table 1. In order to determine the relation between the living lab approach and implementation, studies reporting on success factors for the implementation due to the application of the living lab approach were included. Moreover, all studies were included irrespective of study design. Only full-text articles published in English, German or Dutch were included. The search was restricted to these languages as this covered the expertise of the research team. Studies not concerning living lab approaches in healthcare were excluded, as well as concept papers describing the methodology of living labs without evaluation of implementation. Additionally, commentaries, editorials, letters and books were excluded. Grey literature including conference abstracts and dissertations were also not included as the goal was to assess peer-reviewed literature to explore the relation between the living lab approach and successful implementation. First, two reviewers (NZ and BH) independently checked all retrieved titles and abstracts. Second, full-text articles were screened and selected. Additionally, through the backward snowball method, reference lists of selected articles were checked for possible relevant studies.31 Three papers were assessed through backward snowballing.32–34 Disagreement between the two reviewers (NZ and BH) was resolved until consensus was reached with the help, if needed, of a third reviewer (SJvdB-V).

Table 1

Description of implementation outcomes adapted from Proctor et al28

Extraction of data and analysis

Data extraction was performed to identify which key components of the living lab approach were used and which implementation outcomes were measured in relevant studies. The following information was collected: first author, year, country, target population for innovation, innovation type, aim of the innovation, study design (with detailed explanation), stakeholder type for cocreation or codevelopment of the innovation, programme design and characteristics of the cocreation, purpose of the cocreation programme, outcome of the cocreation programme (process outcomes), living lab key components (user-centric, cocreation, real-life context, test innovation, open innovation), implementation aspects (acceptability, adoption, appropriateness, feasibility, implementation cost, penetration, sustainability), implementation outcome, and relationship between cocreation programme outcome and implementation outcome to determine the relationship between the living lab approach and the implementation. For the implementation aspects the outcomes as recommended by Proctor et al were used.28 Data extraction was divided among two reviewers (NZ and BH) and checked vice versa by the other reviewer. Disagreements were discussed and resolved until consensus was reached. In case of a study referring to another publication for further description of the design or other relevant information, the additional publication was used to add to the data extraction. Data were synthesised through narrative synthesis due to the diverse study designs included and used the outcomes by Proctor et al as a synthesis taxonomy.28 35 36 The goal was to explore the relationship between the cocreation programme outcome and implementation outcome. Textual descriptions were conducted for each included study based on the predefined data extraction sheet. The taxonomy by Proctor et al was used to classify previous research.28 35 After tabulation of data, aspects of each study were textually described.37 Due to the heterogeneity of studies, a pooled effect was not assessed.

Quality assessment

A quality assessment was performed to score the quality of the included studies in terms of methodological rigour of studies based on the tool from Hawker et al38 as it was deemed most appropriate for the heterogeneous articles included in this systematic review (online supplemental file 3). The quality assessment tool was chosen as it covers a variety of research paradigms, which was specifically suitable as we did not discriminate based on study design, but wanted to get a broad picture of different study approaches. Moreover, the quality assessment tool by Hawker et al offered a clear description of the scoring for the following nine categories: abstract and title; introduction and aims; method and data; sampling; data analysis; ethics and bias; findings/results; transferability and generalisability, and implications and usefulness with a maximum score of 36 in total (also see online supplemental file 2). Methodological quality was assessed for each item (4=good; 3=fair; 2=poor; 1=very poor). The quality appraisal by Hawker et al does not propose cut-offs for the quality assessment. Therefore, methodological quality was determined based on the earlier suggested cut-offs by Braithwaite et al. ‘high quality’ (30–36 points), ‘medium quality’ (24–29 points) and ‘low quality’ (9–23 points).39 After applying the Hawker tool to the studies, the categories ‘good, fair, poor and very poor’ were converted into a numerical score by assigning the answers from 1 point (very poor) to 4 points (good). Then, a score was produced for each study with a minimum of 9 points and a maximum of 36 points. The following definitions were used to create the overall quality grades: high quality (30–36 points), medium quality (24–29 points) and low quality (9–24 points). Two reviewers (NZ and BH) performed the quality assessment independently and disagreements were resolved until consensus was reached.

Patient and public involvement

No patients involved.

Results

The final systematic search resulted in N=1173 unique articles for initial screening; N=171 were included for full-text screening of which N=141 were excluded as they did not meet the inclusion criteria. Figure 1 shows the results of the screening process according to the PRISMA diagram and reasons for exclusion. In total N=30 studies were included for data synthesis.

Figure 1

PRISMA flow diagram of study inclusion. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Study characteristics

Table 2 presents the characteristics, living lab components, phase of innovation and implementation outcomes of included studies (N=30). Studies were conducted in 12 different countries. Most studies were conducted in the USA (N=7),40–46 the UK (N=6)47–52 or Canada (N=5).53–57 The majority of studies applied a mixed-methods study design (N=10)42–46 51 53 55 56 58 or a qualitative study design (N=6).48 52 54 57 59 60 For eliciting the stakeholder type, categorisation to sum up the stakeholder types was used.61 According to the so-called 7Ps Framework to identify stakeholders in patient-centred outcomes research, the following stakeholder groups are of interest: patients and the public, providers, purchasers, payers, policy-makers, product makers and principal investigators.61 Most studies involved a combination of patients and public (N=23)40–46 48–55 57–60 62–65 together with providers (N=17)42 44 47–49 51–58 60 63 66 67 in the living lab approach. The combination of patients and public together with providers was used in N=10 of the included studies. Notably, only one study involved policy-makers and none of the included studies involved purchasers, payers or product makers.68 The most common living lab key components identified were cocreation (N=25)40–42 44–51 53 55–60 62–68 and user-centric (N=10).40 41 43 46 48 52 54 60 66 67 A combination of the living lab components cocreation and user-centric were mentioned in N=5 studies.40 46 48 60 67 Three key components of the living lab approach were mentioned in N=5 studies.41 46 52 62 66 The living lab key component open innovation was not mentioned in included studies.

Table 2

Characteristics of included studies (N=30)

Living lab components were mostly applied in the development phase of interventions (N=21).40–42 44 47–49 51 53 55 57–60 62–68 The most mentioned combination was development and implementation phase (N=5)49 57–60 and development and evaluation phase (N=5).42 44 48 49 55 In the studies using cocreation in their living lab approach, studies reported on different stages for their cocreation process (n=17).41 42 46 49 51 54–60 62 64 66 68 69 These stages include, for instance, exploration, ideation and reflection and implementation60 or prototyping and testing.68 The minority of studies using the living lab approach did not discriminate different stages (N=12).40 43–45 47 48 50 52 53 63 65 67 Studies that mentioned cocreation as a living lab component, but did not describe cocreation phases, reported on, for example, a cyclical process with various meetings.48 Two studies made use of formal frameworks for their living lab approach including the Behaviour Change Wheel66 and/or the Theoretical Domains Framework.63 The use of these frameworks shaped the analysis of the qualitative results. For the implementation outcomes, it was possible that more than one outcome was studied and reported on in a single study. The most reported implementation outcome was acceptability (N=22),42–44 46 47 49–55 57–59 62–68 which was often combined with feasibility (N=17).42–46 49 50 52–55 58 59 63 65–67 Remarkably, only one study reported on appropriateness (N=1)51 and one on penetration (N=1).40

Quality assessment

Included studies were classified as either high, medium or low quality. The quality scores ranged from 21 to 35 across the 30 included articles. The results of the quality assessment are presented in table 3. Of the included studies, 12 were considered of high quality.47 48 50 52 53 57 62 63 65–67 Three of the included studies scored low on the quality assessment.44 49 69 The detailed results of the quality assessment can be found in online supplemental file 2.

Table 3

Methodological rigour and quality of included articles

Results

Overview of reported implementation outcomes

In terms of implementation success, most of the studies (N=20/30) reported on a positive implementation outcome40 44–48 50 51 54–56 58 60 62–65 67–69 (see table 4). The results indicated mostly successful implementation in terms of acceptability and feasibility of a healthcare programme or intervention development.44 46 50 51 54 55 58 62 63 65 67 The desired implementation outcomes were not achieved in all studies. For example, one study reported on a positive feasibility outcome, but not on acceptability.53 Although considered feasible the programme did completely transfer the learning into practice and did not lead to significant changes in service delivery compared with before.53 This study, however, still achieved a high quality score according to the quality assessment tool.53 Another study did not describe results regarding acceptability and feasibility but reported on the sustainability of the intervention.49 In this study, sustainability was secured by providing future funding, contracting, a protocol to regularly update the content of the innovation and through facilitating wider spread of the innovation.49 Furthermore, N=3 studies did not achieve their implementation outcome at all.52 57 59 Two studies reported that the intervention was not found feasible.52 59 In one of these, participants in the living lab approach indicated that it was not feasible to apply the living lab approach when also being in charge of resourcing for the execution of the living lab approach.59 The other study still needs to evaluate the feasibility result of the intervention.52 One of the included studies evaluated five of the proposed implementation outcomes.40 They reported that the intervention was more successful in one of the collaborative teams in comparison with the other usual care teams. The characteristics of the successful team are in line with the success factors for the implementation as that team involved middle and senior managers, whereas the other team only involved frontline end-users.40 Moreover, the successful team had a a priori set focus for change, whereas the other team had less of a strict strategy.

Table 4

(A) Synthesis of results in terms of implementation outcomes. (B) Continued synthesis of results in terms of implementation outcomes. (C) Continued synthesis of results in terms of implementation outcomes

The relationship between the living lab approach and implementation

The living lab approach, due to its participatory nature, showed to lead to successful implementation outcomes. The included studies that applied the living lab approach reported on six success factors for the implementation:

  1. Leadership: Leadership in the collaborative and participatory approach of a living lab was seen as crucial to the success of the implementation.44 45 53 54 57 62 68 The presence of senior leadership might contribute to better involvement of end-users53 as well as participants that are already familiar with the intervention to support the implementation.62 In contrast, resistance of senior leaders can hinder the implementation.55

  2. Involvement: Studies made use of participatory techniques to involve end-users in all phases of the living lab. Involvement of end-users was found as a catalyst for the implementation.40 44 48 54 56 57 59 62 68 69 The involvement of end-users early on in the process of a healthcare programme or intervention development can contribute even further to successful implementation outcomes.51 59

  3. Timing: The timing and continuity of participatory living labs may enhance the success of the implementation.40 Unrealistic timeframes for speedy implementations of healthcare innovations can hinder successful implementation as the living lab approach with its participatory nature requires time. However, long durations of implementations can also hinder motivation of participants in the living lab.

  4. Openness: A factor that might contribute to better implementation in a living lab approach, as reported, was openness for change.53 The commitment and willingness of participants in a living lab can support better implementation.53 Openness can also be linked to cultural aspects within organisations54 as well as open communication.64

  5. Organisational support: Many of the studies included a combination of patient and professional involvement. In order for professionals to be able to commit to a participatory living lab, organisational support was found as a success factor.53 54 63 Support from the organisation also includes capacity to support a living lab in terms of staff as well as funding.45 49 58 63

  6. Ownership: A sense of ownership of the healthcare programme or intervention to be implemented with the living lab approach might contribute to more successful implementation outcomes.54 This also includes meeting the needs of end-users for the programme to be seen of added-value42 and shared responsibility.59

These success factors, as identified from the included studies, may contribute to the achievement of desired implementation outcomes when applying the living lab approach.

Discussion

The purpose of this integrative review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations. The results of this review show that cocreation and user-centric were the most applied living lab key components in the included studies. Most studies reported achievement of the desired implementation outcomes. However, most studies only evaluated one or two implementation outcomes. Six success factors for implementation due to the living lab approach were identified: leadership, involvement, timing, openness, organisation support and ownership.

Summary of evidence

Most of the included studies made use of a participatory or collaborative design for their living lab approach. The terms participation or collaboration were mostly used interchangeably while implying the same construct of involving several actors in the process of developing, implementing and evaluating an innovation. In the literature, collaboration is defined as ‘the possibility to gather active contribution from several actors during a creative process’.70 Participation entails the possibility to intervene in the development of an innovation by users in order to meet their needs.70 Living labs are increasingly emerging as they promise to meet the public and policy interest for developing and implementing innovations in collaboration with the public.16 However, the lack of a clear definition of a living lab makes it challenging to determine whether the achievement of two or fewer living lab components in included studies make them living labs as per definition. The proposed definitions in the literature are rather narrative instead of offering clear characteristics that need to be met in a collaborative design to act as a living lab.71 It is, therefore, arguable whether the included studies should be evaluated as full living labs when they were only considering two or fewer living lab components. However, living labs offer the unique possibility to prevent the issues concerning implementation and uptake of innovations in healthcare due to limited public trust or clinical resistance, for example.72 In our review, specifically user-centric and cocreation were the most common components. This result is in line with the proposed definition of a living lab: ‘A living lab is a design research methodology aimed at cocreating innovation through the involvement of aware users in a real-life setting.’71 In our study, we did not elicit whether the number of living lab components used can be linked to the level of success in implementation outcomes. However, we identified six success factors for achieving implementation. These identified success factors are not an exclusive list, but an overview of the most prominent success factors also identified in earlier studies.9 Leadership was mentioned as a facilitator for successful implementation in a collaborative design approach.44 45 53 54 57 62 68 Specifically, senior leadership was identified as an important prerequisite. Besides senior leaders to support the new innovation, however, intermediate or lower level leadership support needs to be present for successful implementation on a larger scale.73 The collaborative character of living labs require committed involvement of all actors, which is also supported in the literature.74 In particular, the involvement of patients who are ultimately the ones benefitting from new innovations in healthcare can help to meet their needs.75 Our study also found that mostly patients and the public were involved in the cocreation process which is in line with an earlier integrative review on living labs.19 In order to achieve involvement, an open culture on an organisational, but also individual level, is required,53 54 whereby the composition of actors needs to be considered.76 The identified success factors may contribute to more favourable implementation success, but causal relations between the success factors and an implementation outcome as suggested by Proctor et al cannot be made. In an earlier systematic review success factors for the implementation of open innovation, which is a component of a living lab, were identified.77 Their results are partly in line with our results including findings as the importance of leadership, network and relationship which is linked to involvement and openness, and culture which is linked to our finding of organisational support and openness.77 Our integrative review on the relation between the living lab approach and implementation success contributes to the existing body of evidence as it identified gaps in current research and bridges knowledge between the fields of implementation science and literature on aspects of living labs.

In terms of implementation outcomes, the most commonly evaluated outcomes were acceptability and feasibility. Of the included studies, N=13 measured acceptability and feasibility together. Most of those studies were of high quality in terms of Hawker scoring. One study achieved the desired implementation outcome, but was of low quality.44 Even though, most studies reported on achieved implementation outcomes, most studies only gave insight into few of the Proctor outcomes and it remains unknown how the implementation scored in terms of the other Proctor outcomes. An earlier systematic review also found that acceptability is the most used outcome as it has a long history in theoretical as well as in the empirical literature.78 For feasibility, measurement instruments are however scarce, which would also explain the rather narrative results found in our review.78 The list of implementation outcomes as proposed by Proctor et al is not an exclusive list, but presents the most common implementation outcomes.28 By evaluating just two outcomes of the implementation, other facets, important to the success of the implementation, might remain unobserved. But, since the majority of included studies did not evaluate the full range of Proctor outcomes, the goal of evaluating all of them might also be unrealistic. Moreover, the measurement of the full range of Proctor outcomes may not be feasible as no standardised measurement instruments exist.78 Moreover, it could also be debated which implementation outcome contributes most to successful implementation. An earlier systematic review assessed factors affecting the implementation of innovations from five levels including structural, organisational, provider, patient and innovation in order to find measures for each of the five levels.79 They found that most measures used to evaluate implementation of innovations concerns organisation, provider and innovation-level measures.79 In our review, we aimed to identify implementation success based on the Proctor et al outcomes irrespective of measurement level.28 By investigating different levels of the implementation, outcomes may have been different.80

Limitations

The goal of this integrative review was to summarise the literature on the relationship between the living lab approach and successful implementation of healthcare innovations. However, this study faced some limitations that need to be discussed. First, the quality assessment was done based on the Hawker et al quality assessment tool.38 This tool was deemed most appropriate as it is comprised of evidence from various perspectives and research methods. However, Hawker et al do not propose cut-offs for the assessment of the overall quality rating of an included study. Therefore, we applied the suggested cut-offs from the literature.39 As the tool in itself does not offer an overall quality rating, it is arguable whether these cut-offs proposed by Braithwaite et al are acceptable. We, therefore, conducted the quality assessment also by calculating the mean and came to similar results when applying cut-offs. For this additional evaluation included studies were assessed for each item from 1 to 4 with 1 indicating a good and 4 a very poor score. The cut-offs that we used for this analysis were chosen as following: .00–1.49=good; 1.50–2.49=fair; 2.50–3.49=poor; 3.50–4.00=very poor. Second, the selection of all studies reporting and evaluating at least one of the Proctor outcomes might have had an impact on the study selection. Studies were included irrespective of study design and measurement instruments used for the success of the implementation. A positive implementation outcome was reported even when included studies only offered narrative results on the impact on implementation outcomes. But since no standardised instrument for the measurement of implementation outcomes exists, we tried to elicit results as closely as possible to the results of included studies. The goal of this integrative review was to summarise all available literature irrespective of study design. Integrative reviews allow for the synthesis of information to gain a broader understanding from both qualitative and quantitative studies.29 We, therefore, believe that we selected all relevant evidence from existing literature. Third, only studies published in English, Dutch and German were included which might have introduced language bias to the study, not presenting all the evidence. However, limiting searches to English-only is still common.81 English is generally perceived to be the universal language of science and research shows no evidence of systematic bias from the use of language restrictions in systematic review-based meta-analyses.82 Not using a language restriction would have led to resource challenges with respect to costs, time, and expertise in non-English languages; however, inclusion would have contributed to ensure generalisability and reduce the risk of bias.81

Future research should focus on evaluating the full range of suggested implementation outcomes by Proctor et al for evaluating whether the living lab approach has impact on all aspects of the implementation of healthcare innovations. Furthermore, studies evaluating the living lab approach compared with a context that did not apply the living lab approach should be conducted to understand the effect or added value of applying the living lab approach. Our study showed that the living lab approach was mainly used in the development phase of healthcare innovations. Future studies should examine whether using the living lab approach in the evaluation of a new healthcare innovation or implementation can also be gainful.

Conclusions

The living lab approach seems to foster collaboration and participation of important actors in the design, development and evaluation of new innovations in healthcare. Six facilitators for successful implementation were found which can help future studies in designing living labs. The evaluation of implementation success needs to be further evaluated as currently no standardised measurement tools exist.

Data availability statement

Data are available on reasonable request. All data generated or analysed during this study are included in this published article and its online supplemental information files.

Ethics statements

Patient consent for publication

References

Supplementary materials

Footnotes

  • Contributors NZ and BH screened eligibility of studies which in case of conflict was dissolved with SJvdB-V. NZ en BH synthesised the data. NZ drafted the manuscript, which was critically revised by SJvdB-V, AGEMdB, CTJH en BH. SJvdB-V as the last author acts as the guarantor for the work and conduct of the study. All authors agreed on the final version of the manuscript.

  • Funding This study was supported by Instituut Gak (Grant number: W.003163).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.