Article Text
PDF
Abstract
Introduction 20 years ago, health professional student placements in rural areas of Australia were identified as an important rural recruitment strategy and funding priority. Since then, there has been a growing body of research investigating the value, impact, barriers and facilitators of student placements in rural areas of Australia. Charles Sturt University, Three Rivers Department of Rural Health, was recently awarded an Australian Government grant to expand their Rural Health Multidisciplinary Training (RHMT) programme, designed to increase multi-disciplinary student placements in rural areas of New South Wales (NSW), Australia. The aim of this study is to determine if the expanded RHMT has a positive social return on investment (SROI).
Methods and analyses The RHMT Programme will expand into the Forbes/Parkes/Lachlan local government areas of NSW where there is a population of 21 004 people, including 3743 First Nations peoples. Data collection includes collecting programme outputs, programme costs and conducting surveys and interviews with students, host organisations, supervisors and community members including First Nations peoples. The SROI will quantify the ‘investment’ required to implement the RHMT programme, as well as the ‘social return’ on the RHMT programme from the student, organisational, supervisor and community perspectives. The SROI will compare the combined cost with the combined return, from a societal perspective, including a 3-year time horizon, with cost data presented in $A 2024/25.
Discussion The findings of this SROI study may influence future Australian government investment in RHMT as a mechanism for supporting rural allied health recruitment and for investing in the local rural economy.
Ethics and dissemination This study has been approved by the Charles Sturt University Human Research Ethics Committee (#H23589) and the Aboriginal Health and Medical Research Council of New South Wales (#2130/23). Results will be disseminated via a peer-review journal publication, as well as conference presentations.
- HEALTH ECONOMICS
- EDUCATION & TRAINING (see Medical Education & Training)
- Health Equity
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
First Nations peoples have been involved in the conceptualisation and design of this expanded Rural Health Multidisciplinary Training (RHMT) Programme evaluation, as the study involves and impacts First Nations peoples.
The 3-year time horizon for the expanded RHMT Programme evaluation provides a strong foundation for a social return on investment analysis (compared with a time horizon of less than 12 months).
The planned data collection is likely to identify currently unknown factors of ‘value’ that stem from the expanded RHMT Programme; to reduce potential bias, these unknown factors of ‘value’ will be independently clarified, quantified and valued during the data collection process.
Introduction
Compared with a traditional cost-effectiveness or cost-benefit analysis, a social return on investment (SROI) analysis takes on a wider economic perspective1 and is defined as ‘a framework for measuring and accounting for the much broader concept of value’.2 A SROI captures the health and non-health benefits by considering the social, economic and environmental costs and benefits and, in doing so, shifts the focus from outputs to impact.1–4 A SROI can be applied to multiple interventions such as those in the health, justice and education settings, including education that pertains to rural student placements for health professionals.1 5–8
20 years ago, health professional student placements in rural and remote areas of Australia were identified as an important rural recruitment strategy and funding priority.9 Since then, there has been a growing body of research investigating the value, impact, barriers and facilitators to student placements in rural and remote areas of Australia.10–13 Rural recruitment pathways have been previously described as vague and interrupted, with an inconsistent return of graduates to the rural setting post-graduation.13 While the financial burden and cumulative commitment required for a rural placement can be prohibitive for some potential students, many who do participate in a rural placement report a positive and supportive rural experience.12 13 In 2020, COVID-19 impacted rural and remote health student placements in Australia, resulting in either cancelled placements or participation in an adapted placement.10
In late 2021, Charles Sturt University, Three Rivers Department of Rural Health, was awarded a Commonwealth Government grant to expand the Rural Health Multidisciplinary Training (RHMT) Programme. The RHMT is designed to expand multi-disciplinary student placements in rural and remote areas of Australia, and it has been previously reported that for every $1 spent under a RHMT Programme in Australia, another $1 is generated in the local economy,11 indicating a positive SROI. The current expansion of the RHMT programme will focus on an increase in health student training through high-quality rural education experiences (both traditional and non-traditional placement types); and additional programmes to ensure students are rural ready and culturally sensitive and engage effectively and collaboratively with rural communities. The aim of this study is to determine if the expanded RHMT has a positive SROI.
Methods and analyses
This study protocol has been reported in accordance with the Consolidated Health Economic Evaluation Reporting Standards 2022, CHEERS 2022; online supplemental additional 1.14 This project has been approved by the Charles Sturt University Human Research Ethics Committee (reference number H23589) and the Aboriginal Health and Medical Research Council of New South Wales (reference number 2130/23). The RHMT Programme will expand into the Forbes, Parkes and Lachlan local government areas (inclusive of Condobolin) of New South Wales (Australia) where there is a population of 21 004 people, including 3743 First Nations peoples.
Supplemental material
An overview of the project is presented in figure 1 (theory of change) and figure 2 (project logic model). In summary, the RHMT programme will aim to deliver the following activites: (a) appoint local clinical educators with demonstrable skills in cultural awareness to lead the programme expansion; (b) empower local health professionals to conduct clinical supervision of health students and offer them support via the Rural Health Education team at Three Rivers; (c) provide 264 weeks of allied health student placements per year in the Lachlan area ensuring that students undertaking a placement complete cultural awareness training, that has been designed and delivered by local First Nations community members and that First Nations students have access to the Charles Sturt mentoring programme; (d) acquire dedicated student accommodation; (e) establish a Rural Allied Health Advisory Committee to provide governance and direction for the RHMT Programme and to support strategies to improve long-term rural workforce recruitment and retention; (f) partner with First Nations Peoples to enable students to develop cultural understanding and cultural responsiveness through cultural safety/rural readiness experiences and training; (g) collaborate with the Parkes Country University Centre to provide support and social connection to health students in the area; and (h) progress an evidence base by contributing to the Three Rivers research agenda via research and higher degree research student appointments, together with clinical-researcher partnership research models.
Project theory of change.
Project logic model.
The aim of this study is to determine if the expanded RHMT has a positive SROI. To address this aim, the primary study question being answered is ‘Does an expanded allied health student training programme in regional New South Wales (Australia), result in a positive social return on investment?’, and this is broken down into the following individual research questions:
What ‘investment’ was required to implement the RHMT Programme?
What ‘return’ on the RHMT programme was achieved from the student perspective?
What ‘return’ on the RHMT programme was achieved from the organisational and supervisor perspective?
What ‘return’ on the RHMT programme was achieved from the community perspective, including First Nations peoples?
Methodological approach
The SROI will combine the actual impact with the potential impact of the RHMT. The actual (or evaluative) impact is the observed impact of the RHMT within the specified time horizon (in this case 1 year). The potential (or forecast) impact is based on the value that will be created if the intended outcomes are achieved over the total time horizon (in this case 3 years).4 Activity data will be compared with pre-determined project targets, with qualitative data collected to provide context. Data collection methods include collecting programme outputs, programme costs and conducting surveys and interviews. Interviews may be conducted as a 1:1 interview or as a small group interview/yarning circle (n=2–6), depending on the preference of the participants. Where consent for recording an interview is provided, interviews will be recorded; however, if consent is not provided, detailed notes will be taken. To increase rigour, all investment and return data will be entered into the freely accessible Excel-based Value Map, developed by Social Value International, and will be analysed within this Value Map.15
Study population and consent
Multidisciplinary health students will be invited to participate in a post placement survey and interview. We will aim to recruit ~20 students.
Inclusion criteria: students aged 18+ years undertaking a health student placement through, or in partnership with, the RHMT Programme. No exclusion criteria. Participant recruitment is via the students’ email. Informed consent is required prior to commencing the survey or participating in an interview.
Host organisations will be invited to participate in a post placement survey and interview. We will aim to recruit ~8 staff from host organisations.
Inclusion criteria: staff employed at the host organisation (aged 18+) who have had contact with the RHMT Programme. No exclusion criteria. Participant recruitment is via workplace emails. Informed consent is required before commencing the survey or participating in an interview.
Clinical supervisors will be invited to participate in a post placement survey and interview. We will aim to recruit ~5 clinical supervisors.
Inclusion criteria: clinical supervisor (aged 18+) who is involved in the RHMT Programme. No exclusion criteria. Participant recruitment is via workplace emails. Informed consent is required before commencing the survey or participating in an interview.
Community members including First Nations peoples will be invited to participate in interviews/yarning circle. We will aim to recruit ~10 community members including at least four First Nations peoples.
Inclusion criteria: community members aged 18+ who are impacted or potentially impacted by the RHMT Programme (aiming for the representation from the Condobolin, Peak Hill, Parkes and Forbes areas). No exclusion criteria. Participant recruitment is via a direct approach by project investigators (not members of the health service) and local First Nations research team member (MA). Informed consent is required before commencing the interview.
Impact of, and response to, participant withdrawal
Following the consent process, participants can withdraw from the project up until the point of the data being de-identified. At this point, it is not possible to remove data.
Setting and location
Rural New South Wales, Australia.
Comparators
There are no comparators.
Perspective
‘Social return’ refers to the impact from the student, host organisation, clinical supervisor, local community and First Nations peoples’ perspective.
Time horizon
3-year project (January 2022 – December 2024).
Discount rate, dead weight, displacement, attribution and drop off
The potential future (or forecast) impacts will have a 3.5% discount rate applied per annum to represent a reduced value on future impacts. In addition to the time-related discount rate, both the actual (or evaluative) impacts and potential (or forecast) impacts will be reviewed for dead weight, displacement, attribution and drop off, using data collected during the projects surveys and interviews, as well as data available in the literature.4 Once the different impacts have been reviewed for dead weight, displacement, attribution and drop off, the determined rates for each will be applied to the social return values. It is expected that there will be different rates applied to the different impacts and that there may be overlap of certain impacts that require the ‘repeat-impact’ to be reduced in value, or valued at $0, for example, the student who intend to work rurally post-graduation potentially overlaps with the community placing value on increased health professional recruitment. Finally, participants will also report the importance of each impact, and while this will not influence the value via a weight, it will establish the importance from the stakeholder’s perspective.
Dead weight indicates that an outcome, or a portion of the outcome, would have occurred anyway, without the RHMT.4 For example, the growth in the local economy was the same for the areas impacted by RHMT, as it was for neighbouring areas that were not impacted by RHMT.
Displacement indicates that another activity did not occur to accommodate the activity of interest.4 For example, a health service did not initiative a new clinic, so the staff could focus on the RHMT.
Attribution indicates that an outcome, or a portion of the outcome, occurred due to a separate intervention.4 For example, if a health service was going to commence an initiative with or without the RHMT, the outcome of the initiative cannot be attributed to the RHMT.
Drop off indicates that while the value of an outcome may last for many years, it may decline in value in the future years.4 For example, the value of ‘enhanced student teamwork’ would decline in value over the coming years if the student goes on to work as a solo private practitioner.
Measurement and valuation of resources and costs (investment), as well as selection, measurement, and valuation of outcomes (return), have been detailed in tables 1 and 2.
Defining the INVESTMENT elements of the social return of investment analysis (to be converted into $A 2024/25)
Defining the RETURN elements of the social return on investment analysis (to be converted into $A 2024/25)
Data collection/gathering
Data will be collected by Charles Sturt University, Three Rivers Department of Rural Health staff, and only de-identified data will be provided to the members of the research team who are external to the university. Data collection/gathering techniques are detailed in online supplemental additional 2–5,and these include Additional file 2: Data Collection Form 1—Multi-disciplinary students (data collection via survey and interviews); Additional file 3: Data Collection Form 2—Host organisation staff and supervisors (data collection via survey and interviews); Additional file 4: Data Collection Form 3—Community Members including First Nations peoples (data collection via interviews); and Additional file 5: Data Collection Form 4—Student Placement Details and Supervisor/Student Activity Logs (data collection via current programme data collection processes).
Supplemental material
Supplemental material
Supplemental material
Supplemental material
Data sovereignty
Data sovereignty,16 as it relates to intellectual property ownership, will be carefully discussed with the study participants before data collection. This is particularly important for the First Nations participants, as non-Indigenous researchers will collect First Nations knowledge and experiences through the interview process. In the context of this study, data sovereignty will include who can access; use and benefit from information that is held within First Nations communities, as well as who has the opportunity and right to define; and use and interpret data relating to First Nations communities. In addition to the data sovereignty defined by the study participants, at a minimum, it is intended that the analysis of the data will include First Nations researcher (MA), First Nations members of the Rural Allied Health Advisory Committee and First Nations stakeholders by explicitly asking if the proposed Social Return of Investment analysis includes things that matter and that are material, and if not, what should be included.
Currency, price date and conversion
All costs will be reported in $A 2024/25. Costs data collected before this time will be inflated by consumer price index via the Reserve Bank of Australia Inflation Calculator.17
Rationale and description of the economic evaluation model
Not applicable as this social return of investment analysis does not include modelling as there are objective cost measures for the economic outcomes.
Analytics and assumptions
A social return of investment is a framework for identifying, measuring and valuing the impact of an activity, and it accounts for the social, economic and environmental values that can come as a result of said activity. It will assign a monetary value to the social, economic and environmental impact.7 8 18 The following social return on investment principles will be followed for the current project: involving stakeholders, understanding what changes, valuing the things that matter, only including what is material, not overclaiming, being transparent and verifying the results.7 8 18 For this study, the compilation of ‘social return’ is inclusive of many diverse areas including learning, connection, capabilities, experience, skills, belonging, referrals, prevention, education, teamwork, employment retention, etc. The investment refers to the Commonwealth Government grant to fund the extended RHMT programme, in addition to the in-kind resources provided by Charles Sturt University.
Following the identification of the key stakeholders, contact will be made with the key stakeholders to introduce the social return of investment methodology. Activities from the project logic model (figure 2) will be costed based on the university record of spending. Where cost data are not available, market rates will be applied. A combination of surveys, 1:1 interviews and small-group interviews will be used to understand what may change, as the impact captured via outputs and outcomes, to ensure the evaluation includes things that matter and that are material and that there is no overclaiming. The impacts will be categorised per the project logic model (figure 2). Quantified outputs and outcomes will have a reference value applied. However, should an output or outcome not have a reference value, we will undergo a suitable process to establish the value. Processes may include techniques such as a Willingness to Pay analysis, the Delphi Technique or a Discrete Choice Experiment. The combined investment cost will be compared with the combined financial return to establish the SROI.
Each impact reported in the data will be defined as an actual (or evaluative) impact or as a potential (or forecast) impact. The results will apply a monetary value to all actual impacts and provide a sub-total for this; then apply a monetary value for all potential impacts and provide a sub-total for this; followed by a combined total for the actual and potential impacts. The investment and return data will be analysed within the Excel-based Value Map developed by Social Value International.15
Characterising heterogeneity, distributional effects and uncertainty
First Nations peoples will be able to self-identify in the surveys and interviews and we will estimate how the results of SROI analysis vary for First Nations peoples, including how the impacts are distributed across this priority group.
To characterise sources of uncertainty in the analysis, each resource/cost (investment) and outcome (return) will be examined with respect to evidential and decision uncertainty.19 Evidential uncertainty includes uncertainty in the sources that contribute to the evidence base (ie, missing or poor-quality data), and decision uncertainty includes uncertainty in the sources that substantially contribute to conclusions drawn from the SROI analysis. The identified sources of uncertainty will be addressed through sensitivity analyses where the source of uncertainty will be adjusted by a factor of 0.75 and 1.50 to understand the impact of that individual source on the SROI findings.
Patient and public involvement statement
Community consultation regarding the programme of work began before the grant application was submitted, and this included 30 letters of support provided from organisations operating in the local community. Extensive consultation and collaboration have continued with these community partners following the grant approval, and this is documented and submitted monthly to the Rural Allied Health Advisory Committee. The research to conduct a social return on investment was discussed with and approved by the Rural Allied Health Advisory Committee, and the research team includes three members who live in the research locale and have been able to continue consultation with community on the research methods proposed. Through formal and informal feedback processes, the research team has received advice from community members about what type of data could and should be collected. The research team adjusted the research methods in response to this feedback. One example is that in conversation with local First Nations community members, it was identified that First Nations peoples would like to be able to self-identify as being a First Nations person and that their data should be included within the whole community dataset, rather than being separate. In line with AH&MRC requirements, the research team submitted a document to this ethics committee outlining the community consultation process, which commenced a year before the ethics application for the research was submitted.
Discussion
Should the expansion of the RHMT produce a positive return of investment, alongside completion of the core project activity including additional health student placements, there will be several tangible benefits to the rural Lachlan region community, in New South Wales. These include recruitment of health professionals to the area, local economy stimulation and a self-reported positive impact on First Nations peoples. The potential positive impact on First Nations peoples includes improving access to health services, improving self-understanding of health conditions and improving health professionals understanding of First Nations cultures. It is, however, noted that in 2022, early stages of the RHMT project implementation plan were limited by the COVID-19 pandemic and regional flooding, which resulted in cancelled or adapted health student placements and reduced the planned data collection period.
With parallels to the current proposed study, a similar initiative and evaluation was completed between 2012 and 2018 for medical students undertaking extended rural health student placements in Queensland, Australia.5 The medical extended rural clinical placement programme reported a positive return of investment, in addition to improved clinical confidence and competence, with greater numbers of medical students planning to work in rural areas post-graduation.5
Should this evaluation demonstrate a positive social return on investment, alongside completion of the core project activity including additional health student placements, national scaling and implementation of the programme should be carefully considered to realise the benefit Australia-wide.
Ethics and dissemination
This study has been approved by the Charles Sturt University Human Research Ethics Committee (reference number H23589) and the Aboriginal Health and Medical Research Council of New South Wales (reference number 2130/23). Plans for dissemination of the project results include publication in a peer-review journal, in addition to being presented at relevant conferences. In addition, at the end of each survey and interview, participants are provided with the opportunity to provide a valid email/postal address so that they can obtain a copy of the project report in 12 months, and a post-programme community event will be hosted by Three Rivers Department of Rural Health to share project outcomes and findings with key stakeholders and community members of the Lachlan region. Plans for sharing and/or future use of data that is not covered in the current ethics application will be subject to a further application for ethical approval.
Ethics statements
Patient consent for publication
Acknowledgments
We would like to acknowledge Dr Jane Havelka, a First Nations researcher and academic who provided critical review of the study protocol. In addition, prior to being awarded funding by the Australian Commonwealth Government for the “Health Workforce Program: Expansion of the Rural Health Multidisciplinary Training Program in More Remote Settings (GO4898)” scheme, this grant proposal underwent a peer review process.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors The guarantor is MN, as noted in the following author’s contributions. Conceptualisation: MN, EG, MA, LF, CL, RM and NB; Data curation: MN, EG, MA, LF and CL; Formal Analysis: EG, LF, CL, RM and NB; Funding acquisition: MN; Investigation: MN, EG, MA, LF and CL; Methodology: MN, EG, MA, LF and CL; Project administration: MN, EG, LF and CL; Resources: MN and EG; Software: RM and NB; Supervision: MN and EG; Validation: MN, EG, MA, LF, CL, RM and NB; Visualisation: LF, CL, RM and NB; Evaluation: Writing – original draft: EG, LF, CL, RM and NB; and Evaluation: writing – review and editing: MN, EG, MA, LF, CL, RM and NB.
Funding This study was funded by the Australian Commonwealth Government for the 2022-2024 “Health Workforce Program: Expansion of the Rural Health Multidisciplinary Training Program in More Remote Settings (GO4898)” scheme.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, conduct, reporting or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.