Background 

Since the emergence of network science last century as a putative solution to the way people are organised, many countries have introduced network forms of governance in their health systems.1 Networks are said to be compatible with principles of devolved governance and decentralisation of services.2–4 It is claimed that they can be conceptualised as ‘postbureaucratic’ entities where the standard bureaucratic forms are hollowed out.3 These forms were implemented to solve ‘wicked’ public problems not amenable to simple solutions,5 6 addressing such issues as how health, social care and education can be more effectively configured and provide improved services to consumers and customers.7 8

In healthcare, many kinds of networks have emerged from loose forums to share information and experiences to middle-range collaborations and partnerships, to tightly defined and integrated organisational forms. No level of healthcare—from macrosystems to microsystems—remains unaffected from deploying, funding or mandating network structures.

Health systems have supported networks for improved relationships, interprofessional service delivery and to give effect to complex policy requirements,9–11 as structures to support quality improvement12 and to support implementation activities.13 Networks also manifest naturally in health systems through workplace, professional and social associations. The term ‘network’ is used extensively in healthcare research and in health services delivery. It is often used as a synonym for ‘partnership’, ‘collaboration’, ‘alliance’ and ‘group’. Or, it may be used more specifically to describe the structure of relationships between people, groups or organisations, joined together through nodes and ties.

Various scholars from differing traditions have researched or discussed how networks operate.14–18 Provan and Kenis list the advantages of network coordination as: enhanced learning, more efficient use of resources, increased capacity to plan for and address complex problems, greater competitiveness and better services for clients and customers.19 Nevertheless, they identify a discrepancy between the acclamation and attention networks receive and the knowledge we have about their overall functioning. The process by which certain network conditions lead to various network-level outcomes is unclear. In spite of the burgeoning literature on networks, most have been descriptive (eg, work by Agranoff and McGuire,20 Huxham and Vangan21 and van Bueren).22 Braithwaite et al contend that this also applies in the health sector, with claims for the effectiveness of networks and communities of practice tending to be theoretical or conceptual rather than empirically well grounded.23 Also, relevant to the measurement of effectiveness of networks is the argument posited by Herman and Renz in the literature on organisational effectiveness, that the measurement of effectiveness of organisations should not be seen as an objective reality.24 Rather, effectiveness is a social construction, an achievement of organisational agents and other stakeholders in convincing each other that an organisation (or network) is pursuing the right objectives in the right way. We also note the wide theory of change literature elaborating effectiveness from the stakeholder perspective, linking actions to outcomes and what an organisation or network can expect.25

The issue of how to evaluate networks has received the intermittent attention of social and behavioural scientists. With the prevalence of network forms, and society’s reliance on and investment in them, we need evaluation models to assess their effectiveness and sustainability. Securing an evaluation framework which could be applied to assessing how well health professional networks perform and how they could be organised to work more effectively is a signature problem.

Evaluating networks to date

There have been multiple reviews of social and organisational networks outside of the health literature.26–30 Borgatti and Foster developed a set of dimensions along which network studies vary, including direction of causality, levels of analysis, explanatory goals and explanatory mechanisms.26 The two latter dimensions were used to construct a 2-by-2 table cross-classifying studies of network consequences into four types: structural social capital, social access to resources, contagion and environmental shaping. The leadership network classification framework of Hoppe and Reinelt categorises four leadership network types, providing a tool for network analysis to assist in strengthening, using and evaluating leadership networks effectively.31 Other authors such as Owen-Smith and Powell argue for going beyond a narrow application of social network analysis (SNA), highlighting the importance of the context and settings of networks.32

According to Provan and Milward, empirical researchers have employed network analysis techniques to understand how agencies coordinate and integrate their activities, but they have tended to emphasise differences in network structures and governance.18 Examinations of the relationship across network structures and activities and measures of effectiveness have been lacking. Turrini et al remark on the fragmentation of the literature on determinants of network effectiveness, with a plurality of definitions, multiple theories, multiple methods and multiple explanations.7 Thus, despite the growth in research on network effectiveness since the early 1990s, many authors note the lack of widely accepted theories about network effectiveness and its determinants.

There have been several systematic reviews of health professional networks and collaboratives.33–35 Cunningham et al’s review found that creating cohesive, collaborative networks (of professionals or agencies) could pay dividends in coordinating care and attending to quality and safety issues and agendas, and positively functioning networks were likely to contribute more broadly to an effective organisational climate.33 However, network vulnerabilities can include cliques, professional and gender homophily and over-reliance on central agencies or individuals. Only one-third of the studies reviewed by Cunningham et al linked network structure with evidence of outcomes. There was a need for further well-designed research examining the relationships between professionals’ network structures and health outcomes in a range of different care settings.

In related work, the systematic review of the public affairs literature by Varda et al found that network structure affects governance, management strategies of administrators and effective collaboration.34 In a systematic review assessing the effectiveness of one type of health network, quality improvement collaboratives (QICs), in improving the quality of care, Schouten et al found that the evidence underlying quality improvements was positive but limited.35 The authors concluded that further knowledge of effectiveness, cost-effectiveness and success factors was crucial to determine the value of QICs. In their review of the literature on QICs, Dückers et al state, ‘the problem is that despite its popularity, the evidence for QIC effectiveness is positive but limited’.36 They call for additional research to test the effectiveness of QICs as a spread agency and to examine their sustainability. Identifying the need to develop methodologies to analyse the impact of integrated healthcare networks in Latin America, Vázquez et al developed an evaluation framework, taking into account context, process and intermediate and final outcomes.37 However, they note that the benefits and risks of such networks have scarcely been evaluated.

All in all, the literature evaluating networks indicates that more work is needed to demonstrate effectiveness, and to identify factors related to improved outcomes. It suggests that no widely accepted evaluation framework has yet emerged. The aim of this paper is to propose a framework to evaluate the effectiveness and sustainability of clinical and health networks for the purpose of rigorous research and policy analysis. Our work is part of a larger study to progress the development of measures and tools for a framework to evaluate health networks and communities of practice.23

Methods

Results: an evaluation framework for health networks

Adding relevant information from systematic reviews33 34 38–40 56 and related network and collaborative research29 52 55 57–60 to Provan and Milward,18 and Turrini et al,7 we embraced a three-level approach in our model: at the community level, the network level and the member level. We accepted the three additional dimensions of internal capacity building performance proposed by Turrini et al,7 and endorsed by Ferlie et al 6 (ability to reach stated goals, capacity for innovation and change, and sustainability and viability). Stakeholders in our studies recognised the temporal aspect of measuring network effectiveness and the need for appropriate measurement for the network’s development stage.41 Therefore, our framework addresses how evaluation can be applied at the three levels proposed by Provan and Milward, while recognising the three additional dimensions of Turrini et al across the network life-cycle, as identified by Mandell and Keast61 and Kenis and Provan.45 We also included network governance as a determinant of network effectiveness as this dimension was recognised in the literature but underplayed in prior evaluation frameworks. In addition, we identified through the literature a range of ‘intervening variables’ to include in our model: leadership, network management, communication strategies and interprofessionalism. Figure 1 presents the evaluation framework.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Evaluation framework for clinical and health networks.

The framework begins by asking the evaluator to assess the network goals by gathering information about its characteristics (type of network), to take into account the role of network synergy or antagony, and to provide a summary of network relationships at three levels. At the community level, the effectiveness of the network can have an impact on services to community members. This is important for policy-makers, and for governments who provide and fund networks, in terms of value for money. In addition, those who represent these clients, such as consumer lobby organisations, must be satisfied by the network’s activities, in addition to the broader public. Networks can also be evaluated at the community level in terms of their contribution to building social capital.62 The cooperation and collaboration among members engendered by the network may benefit the community in ways that would not have been possible if no social capital had been created and maintained.18 Some scholars distinguish between the client and the community level to elaborate the difference between more short-term (better individual health) and long-term results (community resilience or social capital).

At the network level, if a network is to be effective in achieving its objectives, it must become a viable entity and sustainable. For Provan and Milward, to operate effectively individual members must act as a network, and this means incurring organising and transaction costs.18 Some of these costs will be borne by individual members. However, for networks established by government bodies such as in the UK’s NHSs, network establishment and maintenance are led, coordinated and governed by a central, local administrative entity. This entity is referred to as a network broker by Lawless and Moore63 and Mandell.64 As the disseminator of funds and resources, administrator, and coordinator of the network, this type of broker is both the agent of the community and the principal of the network participants: this needs to be factored into the framework. Another measure of network effectiveness is the attraction and retention of members. Network outputs provide a further measure, along with the efficiency of information flows in the network, and the strength of relationships among members across the network.

At the participant or member level, network affiliation may or may not accrue benefits for the member. These may include gaining new knowledge, facilitating collaboration, professional acknowledgement and collegiate support. Through network involvement, members may also change their own practice and effect changes in their organisations.

How these relationships feed into network outputs and outcomes, taking into account the specified intervening variables, must also be considered. Researchers will need to address questions about how to distinguish between outputs (what the network does), outcomes (what immediate effects are visible) and impact (what sustainable changes occur). They may decide to involve those participating in the network in defining effectiveness. From our review of the literature, and from our evaluations of health networks, we suggest that a number of factors should be considered. For networks in the quality improvement area, network outputs and outcomes could include: (1) development of models of care; (2) implementation of models of care; (3) establishment of trials, new services, changed practice, redesign projects; (4) education and training of health professionals, consumers, carers; (5) development of consumer and carer materials; (6) raising the profile of the disease/clinical condition; (7) information dissemination (eg, newsletters, website); (8) journal publications, conference presentations; (9) increased resources for the disease/clinical condition and (10) measurable improvement in health outcomes. Intervening variables could include factors relating to (1) leadership; (2) network management; (3) communication strategies and (4) interprofessionalism. Finally, temporality must be taken into account: an assessment of the network’s short-term, medium-term and long-term effectiveness should be made.

Discussion: how might the framework be applied

In this paper, we present a new framework that can be applied to evaluating how well health networks are performing and to assessing how they can be organised to work more effectively. By reconciling key findings across disciplines, especially from the organisational studies literature on public sector administration and health sector literature, with an expert-informed framework, we have taken a step forward to aid evaluators of health networks. Although researchers in various fields have studied the problem of how to evaluate the effectiveness of networks, this knowledge has not been harnessed across research disciplines to provide a framework that can be applied to evaluate health networks.

This framework can help with providing a shared understanding of key domains and inputs for network evaluation for administrators, funders and researchers working with a range of different types of networks in the health sector. The framework responds to calls to address the complex challenge of evaluating the effectiveness of health networks. With increased international public and private funding of health network structures and reliance on such structures in the 21st century, it is important to have a robust framework to assist in their evaluation. The framework presented here explicitly focuses on key domains associated with network effectiveness, as it identifies organisational and contextual characteristics that we hypothesise are necessary aspects to be taken into account in assessing network effectiveness.

We have attempted to be both comprehensive and flexible in designing our framework. In applying the framework to the evaluation of health networks, consideration needs to be given to the underlying research methods, tools and measures needed to produce an assessment. By way of example in the measurement of performance of one type of health network, integrated healthcare networks, Vázquez et al identify a number of methods and tools, and sources of material, that researchers could apply.37

Approaches to evaluating health networks need to select appropriate methodologies and need to be aware of the types of data that are required for the evaluation. For example, in their review of eight English managed clinical networks, Ferlie et al employed a qualitative approach to assessment.6 65 However, they noted methodological difficulties in evaluating network performance, identifying the need to complement qualitative data with more quantitative data—perhaps clinical outcome data. In another study of English clinical networks, Currie et al used mixed methods (SNA and qualitative fieldwork) to examine the current status and potential for leadership agency and knowledge management to transcend institutional hurdles in network functioning.66

In previous evaluations of clinical and health networks, researchers have employed a range of mixed methods. For example, to evaluate clinical networks, we have used qualitative fieldwork, including stakeholder and member interviews to assess perceptions of the key factors relating to network effectiveness.41 We have also employed an online survey to collect network member demographic, performance assessment and network data (for the purpose of SNA). Similarly, mixed methods have been applied to the evaluation of the role of brokers in a translational cancer network.42 67

The approach we have taken in this study has a number of strengths. We drew from a foundation of several literature reviews that were wide in scope, from an examination of existing network evaluation applications, from our own experience in evaluating a number of different health and clinical networks, and we also reviewed a range of different health sector framework constructions. Experts working in, and with background in, diverse health settings informed the mind-mapping process. Finally, our framework was informed from feedback received from participants in several health network studies that we have conducted.

While recognising these strengths, there are also a number of important limitations to this study some of which suggest the need for future research. Although mind mapping is a useful and flexible research tool, it is reliant on expert opinion. In addition, we acknowledge that all models and frameworks are simplifications of the real world, and thus can fail to capture real life messiness and complexity. Although this framework takes into account recent literature reviews and currently available assessment models there may be other literature and models we have missed. Although we have applied the framework in our health and clinical network evaluations, there is a need for wider testing of the applicability and validity of the framework to show that the various domains (and their interactions) are all important in the evaluation of the effectiveness of health networks.

Conclusion

This paper reports on progress in evaluating networks and presents an evaluation framework designed for health settings. With the high level of societal investment in health networks, it is timely to provide a framework to address the need for evaluation models to assess the effectiveness and sustainability of network forms in the health sector.

We now release this framework as an overarching contribution to network evaluation. We encourage further research to test the utility of the framework, and its reliability and validity, using comparative research designs to assess network effectiveness of different types of networks in different contexts and settings.

The conceptual framework presented here may assist health researchers, social scientists, policy-makers and managers with the evaluation of clinical and health networks in different settings. It is designed to inform our understanding of factors associated with their effectiveness and sustainability.