Participants, interventions and outcomes

Outcomes

The primary outcome of this study is the proportion of inappropriate referrals to Isala Movement Clinic. We define an appropriate referral as one that leads to active treatment, an intervention, that only a specialist can provide, or when a specialist offers more than one consultation with or without direct intervention or additional testing. This outcome will be based on hospital registration (activities for fixed rate per diagnosis) data. All other referrals are potentially superfluous and more variable in nature, so we will arbitrarily define these as inappropriate. We expect the total number of referrals to decrease in time, mostly due to fewer inappropriate referrals.

Data collection time points are before start, after every new step and 13 weeks after finish of the trial, which is 26 weeks after the last cluster has started (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Trial timeline.

We will also evaluate the following secondary outcomes:

1. Costs in primary and secondary care (eg, diagnostics, platform use and routine care data).

2. User and patient experience and satisfaction; inventory of barriers and facilitators related to using and implementing the platform, using surveys as well as interviews and focus groups.

3. Number of e-consultations on the platform and a summary of their basic content.

4. Workload reflected as: Numbers of patients and their diagnostics and treatment of hip, knee and shoulder symptoms in primary care and secondary care. Telephone use by GPs to orthopaedic surgeons.

5. Numbers of appropriate and inappropriate referral letters.

Recruitment, randomisation and blinding

Starting 14 September 2021 until 27 September 2022, in 4 timesteps, all practice owners, locum GPs, salaried GPs and GPs in training working in at least 1 of the 64 participating study practices will sequentially receive access to the platform. GPs will be invited through personal letters, sent by the regional medical coordination centre; an organisation to support the collaboration between GPs and hospital specialists(MCC Klik). This regional group will send monthly reminders in their newsletter as well as email reminders. Also, GPs in training will be informed through their university. Once GPs respond to this invitation, a personal onboarding introduction will be held by the staff of Siilo and the GP will maintain access to the platform until the end of the trial. One month after the introduction with Prisma, GPs will be contacted about their experiences thus far. During the trial, no discontinuation or modification in allocation will take place, but as locums and trainees move around, they will be asked to only use their access to the platform for the practices with access. During the trial, all care as usual is permitted for GPs as they find appropriate.

For data extraction from the EMR after the trial finishes, GPs can choose to sign or withhold approval contract (online supplemental material).

Randomisation/allocation sequence:

MCC Klik composed a list of all eligible practices. Next, they anonymise this list of practices with each practice displayed as a practice number and whether the practice is in the rural or urban area (eg, rural 1, rural 2). This anonymised list is shared with the researchers who perform randomisation stratified on urban/rural location. This randomised list is then sent back to the medical coordination centre who invites the practices following the randomisation scheme. For this, invitation are sent by post and reminders are sent by email. During the trial, the research team has no access to information of the GP practices.

Every 13 weeks, a group of 16 GP practices will be invited to access the platform. Because of the stepped-wedge approach, all GPs are part of the control condition until randomised to the intervention. During the control condition, usual care is provided by direct referral, telephone consultation, or telemedicine consults via a Dutch platform (ZorgDomein) that facilitates information exchange for digital referral letters or e-consultations between care providers. During intervention, care as usual is equally accessible.

The intervention precluded blinding of GPs and specialists. In addition, given that patient knowledge of how communication occurs between GP and specialist is unlikely to affect the number of inappropriate referrals, we decided not to blind patients.

Sample size

We calculated the sample size based on the number of new referrals for hip and knee symptoms (Diagnosis Treatment Combination (DBC) 1701 and 1801) each year (n=4385) to the Isala Movement Clinic. Of these, 68.1% require only an outpatient consultation without surgery or further intervention.

In the Netherlands, GPs refer approximately 14 patients per year for hip and knee symptoms.1 ,11 ,12 Based on an average of four GPs per practice in the study region, we expect approximately 56 referrals per year per practice. Whereas the number of referrals should not change in the control condition, a pilot study in this region showed that teleconsultations for knee symptoms led to a 26% reduction in referral rates (personal communication). Given these estimates, we need to include 348 patients or referrals in a direct comparison to achieve an alpha of 0.05 and a power of 90%. The intra-cluster correlation coefficient is estimated to be 0.2, assuming similar referral patterns within group practices. We will adopt four steps of 13 intervention weeks to improve feasibility.

Finally, the design effect due to cluster randomising is 3.6, and assuming cluster autocorrelation of 0.9, the design effect due to repeated assessments is 0.1913 Thus, we will need to include 18 practices (3.6×0.19 × [348/14]) with 1008 patients for potential referral. This is a quarter of the available practices, making the study feasible. Practices will only be invited to use the Prisma platform after allocation to the intervention; as such, practices may not consent, which could decrease the potential effect. If 60% of practices start, we will need 40 practices and 2240 potential referrals to identify the remaining decrease of 16% (instead of 26%), which remains feasible in this study region.

Patient and public involvement

The patient organisation Zorgbelang Inclusief has offered broad support on patients’ perspectives and will continue to offer their significant experience with the patient interviews and the evaluations of patient experience. They will form a qualitative research team with two patients (one each from client boards in general practice and at the Isala Movement Clinic) and researchers from UMCG to evaluate patient experiences. The qualitative research team will also select patients to serve as coresearchers when preparing interviews and focus groups for the qualitative research.

Data collection and analysis

Data sources, collection and descriptive analysis

For the primary outcome, data collection will consist of routine care data from the hospital. For other outcomes, to evaluate the complete patient care process, we will collect and combine routine care data from the hospital, with routine data from GP electronic records as well as platform activity and qualitative data collection. For the data extraction, the trial codes will be used to anonymise GP and patient details. A data sharing agreement between the research department of UMCG and Isala has been approved. All secondary outcomes will be analysed descriptively.

Patient characteristics will be aggregated to the level of provider. For descriptive characteristics of patients and providers, for categorical variables, we will use frequencies and percentages and for continuous variables we will use mean and SD.

Primary outcome

Quantitative Data collection.routine hospital data (figure 2)

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Quantitative analysis.

After the 1-year study period, anonymised data on patients with orthopaedic symptoms treated in Isala Movement Clinic will be extracted and described per GP (practice) over past years. Data will include basic patient characteristics and routine hospital data for orthopaedic surgery and related specialisms, for example, rheumatology by similarity in patient symptoms.

For calculation of the total number for orthopaedic surgery by GP (practice), we analyse data on the number of registrations for hospital specialist (and peer) consultations and all interventions, per patient. From here, we can differentiate between appropriate and inevitable consultations using data about interventions (eg, surgery or intra-articular injections), further diagnostics and multiple consultations (>1), with the remaining consultations deemed inappropriate, which is our primary outcome. Past hospital referral rates will help to correct for seasonal or historic fluctuations (eg, COVID-19). Data analysis for the primary outcome will take place at six timepoint, starting from baseline, before the intervention and 13 weeks after every new group has started. We have added one extra data collection timepoint after the last group has started to adjust for slow uptake. Each timepoint, the intervention and control group have a different constitution.

Data analysis

For the proportion of inappropriate referrals (primary outcome), we will use a logistic mixed model to estimate the effect of using Prisma compared with usual care. Our dependent variable is referral appropriateness (yes/no), with condition (Prisma vs usual care) as the fixed effect of interest. The models will include random intercepts at the GP practice level and will be adjusted for time as a fixed effect, using dummy variables for the differences between steps. Other covariates to be included as fixed effects are practice size (small/large) and location (city/rural). We will use historic referral patterns if, for example, an external factor such a COVID-19 influences outcomes. Although we do not expect missing data, correctly specified mixed models allow for missing data assuming these are missing at random (MAR).

The primary analysis will be executed on an intention-to-treat basis, and the secondary analysis, on a per-protocol basis. Only practices where one or more doctor has logged on to the system at least once during the intervention period will be included for the secondary analysis.

Secondary outcomes:

Quantitative and Qualitative Data collection: routine hospital data, routine GP data, platform data, referral letters, user questionnaires, interviews and focus groups.figure 2-4

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Subanalysis.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

User and patient experience.

Routine hospital data

We will obtain data on additional testing (eg, laboratory and radiology), requested by the GP either at their own discretion or on behalf of the surgeon, to monitor both the number of tests and the dynamics in this distribution. This could provide more support for potential shift to care provision by GPs as well as an evaluation of the costs which is a secondary outcome.

To indicate any potential workload change, we will also evaluate contact between GPs and specialists by monitoring the number of hospital telephone consultations and requests by GPs for 1 week per month throughout the trial. For this, we endorsed the secretary that receives all the telephone calls for the specialists of Isala Movement Clinic, to count and register the total number of incoming calls for 1 week every month.

Routine GP data

For our secondary outcomes of cost analysis in the study region over time, to register potential shifts in workload and to gather information on patients with musculoskeletal symptoms in primary care, we will evaluate GPs’ electronic patient files. These data will be accessed and extracted through Academic General Practicioner Development Network (AHON), a trusted third-party regional network associated with the UMCG, that facilitates anonymised data sharing among affiliated practices for research and other activities.14 ,15 The AHON database includes patient characteristics, ICPC symptom codes,16 treatment, investigations and letters from specialists.

AHON will start to connect with GP practices for the Zwolle region in the coming year. Before extracting data for this RCT, a pilot will be conducted to develop an algorithm for extracting the large amounts of data we require. The routine care data from the AHON database will be used to evaluate musculoskeletal symptom data registered to International Classification of Primary Care (ICPC) codes for the knee (L15, L78, L90), hip (L13, L75, L89) and shoulder (L08, L91, L92), including general descriptions for joints (L20) and related diseases (L99). Relevant selection criteria identified during the pilot will also be added. All data extraction from GPs files will take place after study completion and will cover the previous 5 years.

Platform activity/app use

Other secondary outcomes are platform use and activity. Characteristics of platform users will be described, differentiating between GPs and GPs in training. Application use will be monitored from the start of platform access to the end of the study. This will include both the time to the first question posted and overall user activity. We will define activity in absolute numbers and create quartiles for posting, and reading or searching. The application system can track login data for posting a case and opening the searchable database.

Our hypothesis is that potential referrals could be prevented not only from asking questions but also from reading and searching for answers. In addition, we will define non-users as GPs who never log in to the app and platform users as those with a minimum of one login when they have access to the platform.

We will calculate the median time spent using the platform, the median response time by specialists and the number of responders per case (with the IQR and SD) for both active and read/search use. Descriptive and correlational analyses will be used to assess the outcome of questions for successful searches in relation to a content analysis of the posted question.

Finally, we will specifically monitor use of the search function under the assumption that a GP will use the same platform for both searching and consultation. If a GP uses the search function and does not post a case, the search will be considered successful and the question resolved, but if a question is posted after using the search function, the search will be considered unsuccessful and the question unresolved.

Platform content

Alongside the login activity, message content of the platform will be analysed using the method described in previous observational research for this type of e-consultation.6 Text from the platform will be transferred to ATLAS.ti17 for coding (using a predefined code tree) and quantitative analysis. We will code for basic patient characteristics, ICPC symptom codes, medical history, question type, answer type, usefulness, and whether questions asked of specialists could have been found in relevant GP guidelines (eg, guidelines of the Dutch College of General Practicioners, NHG). After a case has been discussed on the platform, the GP who posted it can indicate if it was successful, if it was answered adequately, and if they had referred the patient without the platform.

Evaluation of user experience on facilitators and barriers

GP and specialist user experiences will be evaluated through brief surveys conducted before and during platform use. Before the intervention, all randomised users will be invited to complete a short questionnaire about how they usually consult specialists and their expectations of the platform. This questionnaire will be developed using information from user surveys conducted during the Prisma start-up period, together with relevant literature on this topic.6 ,7 ,18 ,19 ,20 The survey will be sent to users of the platform when they start with their onboarding. It is compulsory to answer the survey questions to continue with onboarding. Halfway through the trial, another questionnaire will be sent to GPs to evaluate their experiences of using the platform. If these survey results indicate the need for more detail, additional interviews will be planned (eg, if many comments arise that require more explanation or exploration). By this time, we should also have generated information from recently started GPs as well as GPs with longer access. Given that the Zwolle region takes GP trainees from the UMCG who will have a future role in collaboration and innovation implementation, specific emphasis will be given to 10–15 GP trainees. These will be followed up in a final group discussion at 21–23 months.

Referral letter evaluation to support our primary outcome Figure 3

Alongside the primary and secondary outcome, we will deepen our exploration to gain more knowledge on the referral process from GP to specialist. We will extract data referral letters from GPs to orthopaedic surgeons. Descriptions of the number and direction (address) of referral letters will be complemented by content analysis. First, we will develop a scoring system to evaluate if referral letters support the primary outcome (ie, whether referral is adequate or inadequate).21 ,22 ,23 We have a specific interest in the part of the referral letter that could provide more information on the likelihood of an inappropriate consultation with the orthopaedic surgeon (eg, not having attempted treatments like physiotherapy or medication or having missed standard care steps according to GP guidelines). We will investigate if referral letters illustrated that the GP followed the appropriate standard of care before referral, and if not, the reason given for this decision. After the consultation is finished, we will ask the consulting specialists if they considered the reason for referral (in)appropriate. Finally, we will compare the appropriateness of referral letters from doctors during the intervention and control phases. This process acknowledges that the appropriateness of consultations is determined not only by whether an intervention is necessary but also by whether it helps with decision making when several factors need to be considered.

Cost analysis

Our cost analysis will use a health system perspective, which should provide relevant data for primary and secondary care stakeholders. Although a societal perspective could offer greater detail, the anonymisation and the workload demand on healthcare workers and patients in this study would make questionnaires on production losses impossible. Therefore, difference in costs between intervention with usual care will be calculated including 95% confidence intervals. In both situations, total costs in primary care will be calculated using the GP’s consultation fee and the time investments for consultation and further investigation (counting number of visits). If referred to hospital, the total costs of specialist consultations and procedures will be considered. The cost of platform use in the intervention will also include the e-consultation fees for specialists.3 ,4 ,24 All valuations will use the Dutch cost manual.25

Evaluation of patient experience: interviews and focus groups (figure 4)

A qualitative patient evaluation will take place in the final months of the intervention. As stated under patient and public involvement, a research team will be formed, comprising patient organisation (Zorgbelang Inclusief), researchers, from UMCG as well as patient (representatives) to evaluate patient experience. Because patient (representatives) are involved in the whole process of designing and planning of this research, these patients serve as coresearchers in the interviews and focus groups to answer the qualitative research question ‘what are the facilitators and barriers related to use of the Prisma platform.’ Following the quadruple aim approach should help us to improve the patient experience, with two research questions being relevant to patients: ‘What are the opinions and experiences of patients regarding interdisciplinary e-consultations by GPs for secondary care referral or advice?’ and ‘What are the barriers and facilitators to using this type of consultation from a patient perspective?’

Patient experience will be evaluated in interviews and focus groups,with the interactive inductive and deductive approach to develop different themes and concepts. Previous research on patient experience with e-consultation between GP and specialist has revealed several potential barriers and facilitators for patients.20 ,26 ,27 Together with relevant information from the user surveys, this will inform the development of an initial topic list for patient interviews focusing on ownership and shared decision-making. Zorgbelang Inclusief will provide tape recordings of the interviews and audio data of the focus group meetings, which will be transcribed verbatim and shared with the research group. Two researchers will code the transcriptions separately in Atlas.ti and will check for mutual agreement after both the interviews and focus groups.

To recruit patients without affecting privacy, GPs will be asked to select and invite patients with orthopaedic symptoms at pre-determined points. GPs will be instructed to select the most recent cases; excluding those referred directly to a hospital specialist for an intervention (a consultation is not required). After informed consent is given (patient consent form, order 9), patients will be invited for interview in equal distributions by age, gender and primary health complaint from both the intervention and the usual care steps. The semistructured interviews will be facilitated/moderated by the qualitative research team. Enrolment will continue to saturation. To explore the information and themes raised during interview, we will conduct theoretical sampling within two patient focus groups comprising a maximum of 12 patients, aiming to include patients with a range of characteristics. Zorgbelang Inclusief will facilitate and mediate the focus group discussions and a researcher will observe and support the process. Themes formulated from the patient interviews will form the basis of discussion. The barriers and facilitators related to the use and further implementation of the platform will be formulated and categorised. A small gift certificate will be given to participating patients.

Data management and data monitoring

All data collected from the different locations will be stored according to the data management plan before, during and after the trial in order to protect confidentiality. Following Dutch law, this study does not require a data monitoring committee;this is not a potentially harmful intervention, as the intervention is aimed at communication and sharing knowledge between physicians. GPs will remain fully responsible for the care provided to their patients.