Study design

We will conduct a systematic review alongside a realist review. Realist synthesis is particularly relevant for making sense of context sensitive complex interventions with a heterogeneous evidence base where traditional systematic reviews would often conclude that there is limited or no evidence to inform next steps.18 The conduct of realist reviews is conducive to the study of complex interventions, as simply ‘knowing’ what works reveals very little about the mechanisms that cause desired outcomes and the contexts under which they occur, and can lead to assertions that ‘nothing works’ or ‘results are inconsistent’.19 ,20 The reporting of our reviews will be guided by the PRISMA21 and RAMESES22 criteria. Our protocol was conceived, developed and reviewed by all members of our team, and is registered with PROSPERO, an international register of systematic review protocols (registration number CRD42014014489; http://www.crd.york.ac.uk/PROSPERO).23

Eligibility criteria

We developed our eligibility criteria from our research questions: (1) What is the effectiveness of chronic disease management (CDM) tools addressing one or more high-burden chronic diseases affecting people aged≥65 years? (2) Can the impact of such tools be optimized? For desired outcomes, what are the causal mechanisms and related triggering contexts? We used the following PICOS24 elements to build our eligibility criteria (see online supplementary appendix 1):

Search strategy

We will develop a single search strategy for the systematic review and realist review. With the help of an information specialist, we will search MEDLINE, EMBASE, CINAHL, AgeLine and the Cochrane Library in any language. We are restricting our search to 1990 and onwards as evidence indicates that few multimorbidity studies have been published prior to this,16 and CDM has substantially changed over the past 15 years. We have completed our search in MEDLINE from 1990 to January 2015 (see online supplementary appendix 2). To help identify studies of older adults aged ≥65 years, we will apply a validated age-specific search filter.29 We will also search the grey or difficult to locate literature (ie, conference proceedings, Google Scholar and websites of relevant chronic disease organisations); and scan reference lists of included studies. We will also use the Canadian Agency for Drugs and Technologies in Health (CADTH) grey matter approach, which is a ‘deep-web search tool for evidence-based medicine’.30 This approach uses a checklist to identify international health technology assessment websites, clinical trial registries and health economics resources.30 A second information specialist will validate our search strategy using the peer review process of the PRESS checklist.31 A preliminary search strategy conducted in MEDLINE is available in the online supplementary appendix 2. Once our search strategy is peer reviewed and finalised in MEDLINE, we will adjust and develop this for our other data sources (EMBASE, CINAHL, the Cochrane Library and AgeLine). If during programme theory development and refinement for the realist review we find that we need additional information, we will consult with our information specialist to develop and refine additional searches.

Study selection

We will perform a calibration exercise among reviewers to ensure reliability of screening of titles and abstracts. This will involve two reviewers independently screening 10% of a random sample of citations using our online Synthesi.SR Tool (proprietary online systematic review software developed for our Knowledge Synthesis Center at St Michael's Hospital). We will calculate inter-rater agreement using per cent agreement. We will repeat this exercise until we reach a high level of consistency (at least 90% raw agreement), at which point two reviewers will independently screen titles and abstracts of potentially relevant articles in duplicate (level 1 screening). We will follow a similar calibration procedure to identify potentially relevant articles during level 2 screening (ie, full-text articles). Disagreements at both levels of screening will be resolved through discussions with the research team.

Data collection process

We will develop a data abstraction form and test it with our reviewers on a random sample of 10% of included articles. Once reviewers attain at least 90% raw agreement, two reviewers will independently abstract data on study characteristics, population, setting, CDM tool and its components, outcomes, follow-up, analysis methods, findings and study quality. For the realist review, we will seek data from included sources to iteratively test and refine each section of our initial programme theory. Hence, the data that we need to extract will be informed by our programme theory. For any outcome within a stage of the programme theory, we will seek data that enable us to make inferences about what the mechanism(s) might be and the contexts under which they are triggered. Specifically, sections of text from included sources that support any interpretations we make about the meaning behind the data will be extracted. In other words, if when reading an included source we interpret that a section of text refers to context that is relevant for programme theory development, we will extract that section of text (as well as note its source). Two reviewers will independently map out the relationship between any CMOCs we develop to generate a causative explanation pertaining to the data from the systematic review17 (using NVivo V.10.0 to aid in this process); discrepancies will be resolved through team consensus.

Methodological quality assessment

Study quality will be independently assessed by two reviewers according to study type: the Cochrane Risk of Bias tool for RCTs;32 the Cochrane Effective Practice and Organization of Care (EPOC) tool for non-RCTs, quasi-RCTs (ie, interrupted time series, before-after studies);33 the Newcastle-Ottawa Scale for cohort studies34 and the Critical Appraisal Skills Program (CASP) tool for qualitative studies.35 Additionally, we anticipate that many tools for CDM will be complex interventions (ie, multifaceted with multiple targets), so we will explore their elements to determine which aspect contributes to its impact. To do this, we will extract information about the overall CDM tool or intervention, as well as its specific components or elements (eg, decision support for clinicians, reminders and education for clinicians and patients) using the Template for Intervention Description and Replication (TIDIER) checklist.36 This includes information about the rationale or goal of the elements essential to the intervention; what materials were used in its delivery; who delivered the intervention, and how, where, when and how much; and how well the intervention was delivered as planned.36

Data synthesis

We will perform descriptive summaries of study and appraisal characteristics; and assess the effects of each CDM tools descriptively (eg, data distributions, frequencies, percentages, means, medians, SDs and IQRs). If appropriate, we will perform a meta-analysis to estimate the pooled relative risk (dichotomous outcomes) or mean difference or standardised mean difference (continuous outcomes). Analysis will be performed using the R statistical software, and the results will be presented using forest plots. We will also perform a synthesis of cost data. We will explore the potential sources of statistical, methodological and clinical heterogeneity. Statistical heterogeneity will be assessed using the I² statistic.37 We will consider pooling if heterogeneity among studies is low to moderate (I² <25–50%),37 and a random effects model will be used to account for the observed heterogeneity. We will perform subgroup analyses: by disease, age (65–75; 76–84; 85+), gender, and CDM tools with similar components or similar combinations of components (eg, education+reminder+feedback) and targets (eg, providers, patients). If data are available, we will perform metaregression analyses to formally test if evidence exists for different effects in different subgroups; and assess publication bias using the Egger test.38

CDM interventions and tools are complex (ie, multifaceted with multiple targets), so we will explore their individual elements to determine which aspect contributes to their impact. We will use content analysis to do this: two investigators will review the description of each CDM tool, and independently document its components, by whom and to which target it was delivered (eg, nurse delivers education to patients), at what frequency (eg, twice a week) and duration (eg, 6 months) this was done, and where or in what context (eg, primary care clinic). We will also consider the synthesis of our data according to Wagner's Chronic Care Model (CCM).39 ,40 We will use the CCM to map interventions and their components according to its six organisational/practice change elements for improvement: Healthcare organisation, community resources, self-management support, delivery system design, decision support and clinical information system.39 We may also perform additional and more targeted content analysis of interventions (and their components) identified by our stakeholder team as having potential to inform practice (eg, identified as effective and feasible to implement). As a final step, we will interpret the findings and outline the broader implications for practice and future study.

Source selection, analysis and synthesis—realist review

The review processes for the realist review will be undertaken by two reviewers through regular meetings with the project team, where progress on programme theory development and refinement will be shared and discussed. To assess relevance, the full text of sources included in the systematic review will be read and the following questions asked of the source:41

1. Does this source contain any data that could be interpreted as relevant context, mechanism or outcome for programme theory development?

2. What is the CMOC for this relevant data? In other words, if a section of text is describing relevant context, what might the mechanism be and what outcome does it relate to? Any single source might not contain all the information needed to construct the CMOC. Often sources contain mainly data on context and an outcome and little (if any) details on the mechanism. Thus, from any one source it is often only possible to construct a partial CMOC.

3. How does the (full or partial) CMOC relate to the programme theory? Is there any data in this source to indicate how the CMOC relates to the programme theory? In light of this CMOC and any data on the relationship between this CMOC and the programme theory, are any changes needed in the programme theory? If so, how?

4. Finally and related to (B) and (C) above: How trustworthy are the data used to construct the CMOC? Are they rigorous enough to justify any changes to a CMOC? How trustworthy are the data used to refine the relationships within the programme theory? Are they rigorous enough to justify any changes to the programme theory?

It is at this stage that Pawson’s concept of rigour is used.41 For example, a CMOC based on the opinions expressed in an editorial may be relevant and constructed. The editorial may have a few references, but ultimately it is just the opinion of the authors. The contents of the editorial may be relevant to a CMOC and the programme theory, but caution would be needed before any changes are made to the programme theory to reflect what is in the CMOC until more data of a more rigorous nature are found.

Within a realist review during source selection, extraction, analysis and synthesis, the reviewer is constantly moving between data, to CMOCs and programme theory—that is, moving up and down levels of abstraction. Also, the most common issue that a reviewer will encounter is, any one source often only provides partial ‘bits’ of relevant information to inform (A) to (D) above. The consequence is, to make up the complete ‘picture’, bits from more than one source are frequently needed.