Article Text
PDF
Abstract
Introduction Unlike other forms of evaluation, social return on investment (SROI) methodology offers a way of placing values on personal, social and community outcomes, not just economic outcomes. Developed in 2000, there have been calls for greater academic involvement in development of SROI, which to date has been more typically implemented in-house or by consultants. This protocol describes a systematic review of SROI analysis conducted on health and social care programmes which represent a significant sector of social enterprise internationally. The aims of the systematic review are to (1) identify the extent to which academics have adopted SROI methodology, (2) how academics have interpreted, used and developed SROI methodology and (3) to assess the quality of studies published under peer review.
Methods and analysis The systematic review will include peer-reviewed studies since 2000 published in English. Search terms will be ‘social return on investment’ or ‘SROI’. Health and social care interventions will be identified in the initial screening given the proliferation of possible key words in these areas. Databases to be searched include Web of Science, Scopus, Medline, Social Care Online and National Institute for Health and Care Excellence. Two reviewers will independently conduct initial screening based on titles and abstracts against the inclusion criteria. Data extracted will include date of intervention, country, study design, aim of intervention/programme, participants and setting, health and social care measures used, and SROI ratio. The quality of studies will be assessed by two reviewers using a SROI quality framework designed for the purpose of this study.
Ethics and dissemination The systematic review will review existing published academic literature; as such, ethics approval is not required for this study. A paper of the systematic review will be submitted to a peer-reviewed journal.
PROSPERO registration number CRD42018080195.
- health economics
- social care
- social impact
- social return on investment
- sroi
- social care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A strength of our study is that peer-reviewed literature will be compared with other peer-reviewed literature.
We expect that we will be able to comment on the challenges of presenting social return on investment (SROI) findings within the confines of peer-reviewed word counts (as opposed to in reports without such word count restrictions, which has been the norm to date) and how academics establish rigour within such word limitations.
Another strength of our study is that we have developed a quality framework for SROI studies which this systematic review will use for the first time.
We anticipate that this will be a useful tool to assist with standardising reporting of SROI studies in the academic literature.
An anticipated limitation of the study is that academics have only recently started to adopt SROI methodology and, as such, we may find few qualifying studies.
Background
In the last decade, many developed countries have seen a growth in the number of private and social enterprises operating in the health and social care sectors.1–3 For example, in the UK, there are estimated to be 471 000 social enterprises employing approximately 1.44 million people.4 In Australia there are estimated to be approximately 20 000 social enterprises.5 The significance and growth of the health and social care sector within social enterprise has been noted internationally. For example, health and social care social enterprises are estimated to be 22.2% of all social enterprises in Australia, 30% of those in Pakistan and Ghana, and 20% of social enterprises in Vietnam (healthcare only).6–9
Social enterprise stakeholders are interested in measuring outcomes in order to ensure programme objectives are being achieved and that funds are being allocated to maximise social impact.10 Traditional forms of economic evaluation have focused on financial benefits and outcomes, such as savings from decreased welfare payments in the case of interventions to support welfare recipients into employment11 12; however, social return on investment (SROI) analysis offers a methodology that seeks to evaluate and place proxy values on personal, social and community outcomes, as well as more broadly capturing social impact at the societal level.13 14 Establishing social impact using SROI methodology can also assist organisations in acquiring resources, reinforcing mission and leading to valuable organisational learning.15 16 Initially developed in 2000, SROI is a methodology with acknowledged strengths and challenges that is still developing.2 10 15–17 There have been calls for academics to use and develop the SROI methodology which has most commonly been implemented by consultants.10 13 18 This paper describes a protocol for conducting a systematic review of health and social care-focused SROI studies published in the academic literature. This protocol outlines SROI methodology and the extent to which the methodology has been used to date. We then outline the objectives of our systematic review and our search strategy.
Social return on investment methodology
SROI analysis was first conceptualised in the USA in 200019 and later tested and refined in the UK by the new economics foundation.2 20 21 Some key features of SROI that differentiate the approach from that of other forms of economic evaluation (eg, cost–utility and cost-effectiveness analysis, routinely applied in health economics) are in the creation of a theory of change which captures the associations between inputs, outputs and outcomes, the engagement of stakeholders and in valuing outcomes which are not typically measured in other types of economic evaluation (see Banke-Thomas et al 17 for a comprehensive comparison of cost-effectiveness analysis, cost–utility analysis, cost–benefit analysis and SROI).
In SROI methodology, financial proxy values are identified for each outcome in order to provide estimations of the social value generated by an intervention. For example, the cost of psychological counselling services could be used as a financial proxy for mental health and well-being. SROI is expressed as a ratio of the discounted value of benefits (social value) divided by total investment. Discounts to social value are calculated on the basis of what would have happened without the intervention (deadweight), what activities were displaced by the intervention (displacement), who else has contributed to the outcomes aside from the organisation under study (attribution) and whether experience of the outcomes declines over time (drop off). Costs incurred and benefits realised over different time periods are made comparable using discounting to calculate net present value.14 As an example, an SROI ratio of 3:1 illustrates that $3 of social value was created for each dollar invested once appropriate discounts have been made.
Use of the methodology has been particularly widespread in the UK, where it has been promoted by the new economics foundation, and by the UK-based SROI network.13 Indeed, the SROI Network published a guide to conducting SROI analysis, which is freely available in the public domain in 10 different languages14 (the organisation was renamed Social Value UK in 2015). Adoption of SROI methodology among social enterprises has been particularly encouraged in the UK by the establishment of the Social Enterprise Investment Fund, which provides both business and financial support to social enterprises seeking to measure the social impact of their programmes.2 Use of SROI methodology has also been observed outside the UK—particularly Australia, the USA and Canada—which in their meta-analysis, Krlev et al 13 attributed to the alignment of the methodological framework with the Anglo-Saxon performance orientation.
However, it is important that there is some means of assessing the quality of SROI studies, especially given that studies can be undertaken with differing levels of expertise and with different levels of resources.10 16 To date, many studies internationally have been undertaken in-house by organisations or by SROI consultants.10 13 18 As such, SROI studies may or may not be in the public domain in part or in full10 13 22 which can potentially limit learning from previous SROI studies, as well as limit the ongoing development of the methodology. SROI studies designed and conducted by academics under the peer-review process have thus far been few. In their meta-analysis in 2012, Krlev et al 13 reported that just 1% of studies included were from the peer-reviewed literature. However, in a systematic review by Banke-Thomas et al 17 focusing on public health SROI studies, the authors identified that 10% of studies to 2015 were peer reviewed. This difference between general SROI studies and those in health may reflect a higher need among healthcare professionals to validate their SROI analysis to ensure outputs align with evidential standards of practice expected in the healthcare sector. Several authors have identified the need for academics to engage with and develop SROI methodology as well as to standardise practice.13 16–18
There has been one systematic review and one meta-analysis of relevance to our own systematic review. Krlev et al 13 conducted a meta-analysis of all types of SROI studies between 2002 and 2012. This study identified the scope of SROI studies conducted to date as well as underlying trends in the growth of SROI. Though this study was scholarly and conducted by academics, it was not published under peer review. Furthermore, this study was not replicable as the authors accessed several of the studies included via their personal and professional networks. These studies were therefore not in the public domain and not easily accessible to other reviewers. Krlev et al 13 assessed the quality of the SROI studies included in their meta-analysis. To this end the authors outlined a 12-point quality framework, which they developed for the purpose of their meta-analysis. These 12 points related to five main areas: (1) transparency about why SROI was chosen, (2) documentation of the analysis, (3) study design, (4) precision of the analysis and (5) reflection of the results. The authors developed their own quality framework as there was nothing in the public domain relating to the assessment of the quality of SROI studies at that time.
Another study of note was a systematic review conducted by Banke-Thomas et al 17 on SROI studies of public health interventions, including both peer-reviewed and grey literature. The primary focus of this systematic review was on value for money, the overall quality of studies and lessons learnt. In terms of assessing the quality of SROI studies, Banke-Thomas et al 17 used the same quality framework developed by Krlev et al,13 noting that this was the only quality framework for SROI studies available the public domain. The authors identified a need to develop a more robust quality assessment framework for SROI studies in health.17
The current study
In this systematic review, we will add to the academic literature with a detailed exploration of (1) the extent to which academics have adopted SROI methodology, that is, has there been an increase in the number of peer-reviewed SROI papers; (2) how academics have interpreted, used and developed SROI methodology; and (3) how academics have reported SROI studies using a quality review designed for the purpose. As such, unlike the meta-analysis13 and systematic review17 presented above, this study seeks to compare peer-reviewed literature. This is important as, unlike grey literature, peer-reviewed literature is typically subject to higher evidential standards as well as strict word count limitations. How SROI studies establish academic rigour under peer review is important for learning from previously conducted studies as well as for establishing standardised reporting practice.
Methods
Design
This review will be conducted using Preferred Reporting Items for Systematic Review and Meta-Analyses Protocols guidelines and checklist.23 The protocol is registered with the International Prospective Register of Systematic Reviews (PROSPERO) (number CRD42018080195).
Inclusion criteria
The focus of this review is the SROI studies in health and social care settings; as such any age group or population will be included if the intervention or programme is health and/or social care related.
Exclusion criteria
Publications which are not peer-reviewed, those pertaining to other cost-effectiveness measures (such as cost effectiveness analysis or cost–utility analysis), conference abstracts and papers not published in English will be excluded.
Types of studies
All empirical study types that measure SROI for a health and/or social care intervention.
Search strategy
Given the focus of the study and intent to only identify studies that used SROI methodology, the key word search will be limited to ‘social return on investment’ and ‘SROI’ and will be based on full-text electronic searches. Due to there being numerous keywords variations for health and social care, returned items will be subject to independent title and abstract search by two reviewers to determine relevancy. A third reviewer will resolve any conflicts following this stage of the screening.
Searches will be limited to papers published after the year 2000 to mirror the time frames of SROI methodology development. Paper published up to 1 October 2018 will be included. Due to the multidisciplinary nature of SROI, electronic searches will be performed on the following databases; Web of Science, Scopus, CINAHL, Econlit, Medline, PsychINFO, Embase, Emerald, Social Care Online and the National Institute for Health and Care Excellence (online supplementary appendix I).
Supplementary file 1
The reference list of all relevant papers will be searched for studies that have not been identified electronically. A citation search will also be performed on Google Scholar, and Scopus and related articles identified.
Screening and data extraction
Search results will be stored in Covidence systematic review software24 and duplicates removed. Titles and abstracts will be screened by two reviewers independently against inclusion criteria to ensure internal consensus and reduce the risk of bias. Full-text manuscripts will be obtained and filed in Covidence where each will be independently reviewed by two reviewers. Any disagreements will be considered by a third reviewer and then resolved by consensus.
In order to be able to describe the qualifying studies, data will be extracted on the following categories: author, date of publication, country, SROI type (evaluative or forecast), participants and setting, method of analysis, SROI ratio and outcomes.
Quality assessment
A SROI quality framework was developed for the purpose of the review by our multidisciplinary team headed by a senior Health Economist (supplementary appendix II) as, based on the discussion above, there was no established peer reviewed quality framework to critically appraise published SROI studies. The resources used to develop the first draft of the quality assessment framework were the 12-point framework developed by Krlev et al,13 the Drummond economic evaluation checklist25 which has been used extensively for healthcare-based evaluations, the McMaster Critical Review form26 which has been used extensively to assess the quality of qualitative studies in numerous domains including health and social care27–29 and the Guide to Social Return on Investment by Nicholls et al.14 This guide was published by the Office for the Third Sector in the UK Cabinet Office and has been the framework extensively adopted internationally.
Supplementary file 2
The first draft consisted of 23 questions in eight areas: (1) research question, (2) reason for using SROI, (3) scope, (4) theory of change/impact map, (5) study design, (6) analysis, (7) embedding findings and (8) assurance. The draft framework was trialled on five SROI studies identified in an initial literature research by two independent reviewers. Following this trial, the quality framework was reviewed by all authors, and amendments made to simplify and clarify some of the questions. The second draft was trialled on three randomly chosen papers from the initial five of the first trial by an academic colleague outside the research team. The quality framework then underwent a final review by all authors. Amendments included making the language more consistent with the SROI approach and adding additional explanation under some of the questions. The quality framework has been subsequently reviewed and amended as part of the peer-review process for publishing this systematic review protocol. Following feedback from reviewers questions relating to embedding findings and assurance were removed as not relevant to peer-reviewed SROI studies reducing the quality framework to 21 questions in six areas. In the systematic review, quality assessment of all selected studies will be conducted independently by two reviewers. Each of the 21 items can be scored according to four unique response categories: yes, no, not clear and not applicable. If data are not reported, it will be scored as a ‘no’, and if data are inadequately reported it will be scored as ‘not clear’. If an aspect of the quality framework is not relevant to a particular study, for example, the intervention does not include capital costs, that it will be marked as ‘not applicable’. If all items are relevant, the final score will be out of 21, though we expected there will be some papers where this will not be the case and therefore the score may be out of 19 or 20, for example.
Data synthesis
Data will be synthesised to address the three stated objectives of the systematic review. To address objective 1, the number of qualifying studies will be compared with findings from a previous systematic review of public health peer-reviewed and grey literature17 as an indication of whether there has been an increase in SROI studies since 2015. Data on the interpretation, use and development of SROI methodology (objective 2) will be determined by a qualitative review of the methodology adopted compared with the methodology outlined in the SROI Network’s Guide to Social Return on Investment,14 which is extensively cited in both peer-reviewed and grey literature studies. In addressing this objective, we also expect to draw on SROI methodological commentaries.15 16
Finally, findings of the quality review (objective 3) will be reported in table format and synthesised into a narrative format. Due to the anticipated heterogeneous nature of the results, meta-analysis is not likely. However, we expect to report and comment on meta-biases within published SROI studies.
Amendments to the protocol will be avoided; however, if the search strategy is identified to require additional rigour or adaptation, action will only occur with the consensus of the full research team. All decision points will be documented reflexively.
Patient and public involvement
This paper describes a protocol for a systematic review and therefore there is no direct patient or public involvement. However, participants with disability in the broader research project have been involved since the inception and have contributed to the objectives outlined in this systematic review.
Ethics and dissemination
This review aims to provide a systematic review of existing published academic literature; as such, ethics approval is not required for this study. This systematic review is registered with the International Prospective Register of Systematic Reviews (PROSPERO) (registration number: CRD42018080195). The methods and findings from this systematic review will be disseminated as a peer-reviewed journal article.
Acknowledgments
The authors thank Kate Kennedy for conducting the second trial of the SROI quality framework.
Footnotes
Contributors CLH, AB and JR conceptualised this study. CLH developed the quality framework. CLH and AB wrote the first draft. JR, SG-H and SG agreed the methodology, provided feedback on the quality framework, provided feedback on the first draft of this protocol paper and agreed the final draft. CLH and AB will lead the systematic review. Amendments to this protocol were primary conducted by CLH and JR. All authors reviewed and approved final amendments.
Funding This work was supported by The Lifetime Support Authority, South Australian Government grant number GA00040. The funder was not involved in developing this protocol.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.