Global use of electronic patient-reported outcome systems in nephrology: a mixed methods study

Objectives The use of electronic patient-reported outcome (ePRO) systems to support the management of patients with chronic kidney disease is increasing. This mixed-methods study aimed to comprehensively identify existing and developing ePRO systems, used in nephrology settings globally, ascertaining key characteristics and factors for successful implementation. Study design ePRO systems and developers were identified through a scoping review of the literature and contact with field experts. Developers were invited to participate in a structured survey, to summarise key system characteristics including: (1) system objectives, (2) population, (3) PRO measures used, (4) level of automation, (5) reporting, (6) integration into workflow and (7) links to electronic health records/national registries. Subsequent semistructured interviews were conducted to explore responses. Setting and participants Eligible systems included those being developed or used in nephrology settings to assess ePROs and summarise results to care providers. System developers included those with a key responsibility for aspects of the design, development or implementation of an eligible system. Analytical approach Structured survey data were summarised using descriptive statistics. Interview transcripts were analysed using Codebook Thematic Analysis using domains from the Consolidated Framework for Implementation Research. Results Fifteen unique ePRO systems were identified across seven countries; 10 system developers completed the structured survey and 7 participated in semistructured interviews. Despite system heterogeneity, reported features required for effective implementation included early and sustained patient involvement, clinician champions and expanding existing electronic platforms to integrate ePROs. Systems demonstrated several common features, with the majority being implemented within research settings, thereby affecting system implementation readiness for real-world application. Conclusions There has been considerable research investment in ePRO systems. The findings of this study outline key system features and factors to support the successful implementation of ePROs in routine kidney care.Cite Now


INTRODUCTION
Patients at all stages of chronic kidney disease (CKD), and particularly those undertaking kidney replacement therapy, experience a high symptom burden and often report diminished health-related quality of life (HRQOL). [1][2][3] However, some symptoms remain under-recognised and unalleviated, leading to increased patient burden. 4 HRQOL is not only an important outcome in itself but also associated with clinical outcomes such as healthcare utilisation and mortality. 5 Symptom burden and impact on HRQOL can be assessed using patient-reported outcomes (PROs), defined as 'any report of the status of a patient's health condition that comes directly from the patient, without interpretation of the patient's response by a clinician or anyone'. 6

STRENGTHS AND LIMITATIONS OF THIS STUDY
⇒ The mixed-methods approach offered a greater understanding of the barriers and facilitators of electronic patient-reported outcome (ePROs) implementation, supplementing survey findings on key system characteristics with in-depth interview data from ePROs system developers. ⇒ The electronic survey and interview topic guide were developed with input from stakeholders, including clinicians, PROs methodologists and patients. ⇒ The review was restricted to English-language publications of ePRO systems. Therefore, it is possible that potentially relevant publications published in other languages were excluded. ⇒ Purposive sampling methods were used due to the limited numbers of people who could act as a primary data source, that is, survey and interview participants could have a clinical or non-clinical role but must have carried key responsibility for aspects of design, development or implementation of an eligible ePRO system. ⇒ It is possible that other ePRO systems that could be used in nephrology settings were not identified by this study.

Open access
PROs assessing the impact of CKD and associated treatments can be collected electronically (electronic PROs; ePROs). With the increase in telemedicine, accelerated by the COVID-19 pandemic, 7 8 the collection of ePROs has increasing significance. Web-based systems and 'bring your own device' schemes offer patients new ways to report symptoms, function and HRQOL; they can report PROs in 'real-time' from home or in clinic prior to their consultation. Such use could assist in the management of symptoms, while ensuring that healthcare resources are used effectively to maximise impact. 9 There is increasing evidence, particularly from oncology, that ePROs are cost-effective, 10 can facilitate shared decision-making, promote self-management, improve symptoms and HRQOL, reduce unplanned hospitalisation and enhance long-term outcomes including survival. [11][12][13][14] In contrast to paper-based collection, ePRO systems offer enhanced ease of use and reduced burden, increased user satisfaction (patient and clinician) and lower rates of missing dataf. [15][16][17][18] Evidence around implementation of ePROs in nephrology is growing. Studies from North America demonstrate the feasibility of ePRO collection in haemodialysis (HD) settings, 16 19 20 while the Australian Symptom monitoring With Feedback Trial 21 and Canadian EMPATHY studies 22 are cluster randomised controlled trials exploring the usefulness of integrating PRO assessments in the clinical management of patients receiving HD. In advanced CKD populations, the use of an electronic Patient-Reported Outcome Measure in the Management of Patients with Advanced CKD (RePROM) study in the UK piloted the use, and established feasibility, of an ePRO for remote symptom monitoring in real time. 9 23 The AmbuFlex telepro system is used in practice to manage renal outpatient follow-up in Denmark and as a complimentary tool in consultations 24 25 and is currently being evaluated through the PRO-KID trial: a non-inferiority pragmatic randomised controlled trial investigating the effectiveness of the quality of care, use of resources and patient outcomes associated with PRO-based follow-up in patients with CKD. 24 With this growth of ePRO system research and implementation in CKD globally, there is an opportunity to explore the key characteristics associated with successful implementation to support widespread adoption. The objectives of this study were to 1 comprehensively identify ePRO systems designed specifically for use with CKD populations, including those under development, 2 to explore system characteristics, including methods of administration, levels of integration into existing workflow and electronic health records (EHR) and the reporting of ePROs to manage patient safety.
The aim is being to provide a comprehensive classification of core factors, which contribute to successful implementation of ePROs in nephrology, including common facilitators and barriers.

STUDY DESIGN Methods
Predicated on the research paradigm of pragmatism, with a focus on analysis of study data through the lens of its practical consequences and actionable knowledge, 26 this three-phased study used mixed methods: Phase 1 comprised a scoping review identifying ePRO systems and corresponding developers. 27 Eligible systems were those being developed, used or under study that were designed for or reconfigured specifically for use in adult nephrology settings but excluding those developed for the management of acute kidney injury and paediatric CKD populations. Systems were also excluded if PROs were not assessed electronically and did not provide a summary of the patients' responses to their care provider for use in routine care, that is, were being used solely to collect research or population-level data or to assess the effectiveness of an intervention.
Eligible systems from any country were included. A system developer could have a clinical or non-clinical role but must have carried key responsibility for aspects of design, development or implementation of an eligible ePRO system.
Systems were identified through publications in English, including conference abstracts and grey literature; adapting a previously applied search strategy is used to investigate nephrology PRO measures. 28 Databases (OVID Medline, EMBASE, APA Psychinfo and CINAHL) were searched from dates of inception to 15 December 2021 (see online supplemental file 1, for example, search strategy) followed by ongoing 'pearl' and 'snowballing' methodology, that is, searching from known key references and checking reference lists. [29][30][31] Field experts were also consulted. Data from identified publications were used to create a structured survey exploring key system characteristics (see online supplemental file 2), and to identify system developers.
Phase 2 involved survey administration via an online survey platform Smart Survey (www.smartsurvey.co.uk), which was piloted prior to use. System developers were invited to participate via email. The survey focused on system design and software features, integration of e-PRO collection and reporting in clinical care.
Phase 3: optional follow-up semistructured interviews were undertaken online via videoconferencing software with survey participants, or a suitable team member nominated by the original survey respondent. The purpose of these interviews was to expand on survey responses and allow more detailed system/organisational enquiry. All follow-up interviews used a piloted study-specific topic guide (see online supplemental file 3). Participants were given the opportunity to check and review transcripts, to comment and avoid disclosure of any proprietary information. Interviews were conducted by NEA, a renal research nurse with previous experience of conducting qualitative research. 32 This study was undertaken as part of a Doctoral Research Fellowship, which was disclosed in the participant information sheet and during the consent Open access Table 1 Overview of ePRO systems (see online supplemental file 4 for additional system data) SMaRRT-HD 20 43 USA In-centre HD Unknown-feasibility study reported SMaRRT-HD -Study-specific measure

Open access
process. Some interview participants were previously known to the interviewer, as experts in the field of study.
Maintenance of a reflective diary, memos and field notes, along with discussions with the study management team, were used to minimise the influence of any prior relationships on data analysis.

Open access
Analysis Data derived from the phase 1 review were abstracted and charted (see table 1), and descriptive statistics from the phase 2 structured survey were tabulated (see tables 2-4). Analysis of phase 3 interview data was undertaken using Codebook Thematic Analysis 33 34 and the domains from the Consolidated Framework for Implementation Research (CFIR) (figure 1). Primary data analysis was conducted by lead author (NEA) with a second investigator reviewing coding (CM) for consistency and appropriateness. The CFIR is a determinant framework that can be used to identify and delineate contextual factors (ie, barriers or facilitators) that influence the outcome of implementation efforts. 35 36 Computer-Assisted Qualitative Data Analysis Software CASDAQ (QSR NVIVO V.12 37 ) was used to facilitate qualitative data analysis. Thematic analysis using a framework approach was chosen to systematically identify and analyse patterns of meaning within data, with the aim of highlighting the most salient features. 33 The CFIR was used to gain insight into the overall effectiveness of the ePRO systems and associated implementation strategies. It is possible to rate constructs using the CFIR to undertake organisational comparison via Qualitative Comparative Analysis. 38 Due to the heterogeneity of systems, their context and stage of development, there was a danger of oversimplifying complex, dynamic descriptions of implementation processes and contexts, so this aggregated data approach was not taken. Timing of assessment* Two or more sources for PRO selection (patient/provider)

Open access
Due to time constraints, participant checking of findings was undertaken by one participant only.

Patient and public involvement
Patient partners were involved in the design of this study and coauthored the final manuscript; where they specifically highlighted the importance of ensuring that ePROs do not exacerbate existing health inequalities due to digital inequalities (lack of digital access or skills).

RESULTS
The scoping review identified 14 ePRO systems across seven countries from 43 papers and one developing system identified by field experts, totalling 15 systems: Canada n=4, USA n=3, United Kingdom n=4, Denmark n=1, France n=1, The Netherlands n=1, Australia n=1 (table 1). Online supplemental file 1 includes the Preferred Reporting Items for Systematic Reviews and Meta Analyses Extension for Scoping Reviews (PRISMA ScR) diagram.
Ten system developers responded to the survey (66% response rate). System developers held varying academic and clinical academic research posts, all demonstrating outcomes methodology expertise. Key responsibilities ranged from funding acquisition and stakeholder engagement, to overseeing system development, piloting   Open access and evaluation. Seven developers agreed to a follow-up interview: one expert delegated this task to a suitable member of their study team; three declined. Reasons for not taking part in the follow-up interview were not collected. Interviews ranged from 42 to 82 min in length (mean 63 min). Table 2 outlines characteristics of surveyed systems and system objectives. The most common primary objective was symptom management associated with CKD and/or renal replacement therapy by HD. However, systems were implemented across kidney disease stages and treatment modalities (see table 1 and online supplemental file 4 for additional system data). Aggregate uses of ePROs for benchmarking or commissioning of services were not common; one system collected satisfaction with care data 39 and two others assessed quality of care. 21 40 There was variation in funding source; academic (40%), government (30%) and charitable funding (30%), with 70% of systems being developed in the last 5 years. Half of the systems were developed in-house by local informatics teams, with the rest reporting collaborations across sectors with registries and technology companies. Systems allowed ePROs to be completed in different settings; the most common setting for completion was hospital clinic (80%). Only one system could be accessed by primary care. 40 One surveyed system, Ambuflex, had been implemented into routine practice but was undergoing additional evaluation; 25 all other systems were at varying stages of study with most having conducted pilot/feasibility testing. Two identified systems were being evaluated within a cluster randomised controlled trial, 21 22 while another was being evaluated in a codesigned demonstration trial. 40 Tablet was the most popular mode of administration. Two systems offered a clinic-based kiosk facility. 25 41 Training was available or planned for clinical staff in all systems but focused largely on system access and functionality. Seven systems had developed training materials for patients and four for administrative staff. Four systems had patient education material administered within the system, only one of these linked self-management guidelines to scores. 9 Only 60% of systems were currently linked with the EHR.

Key system characteristics
All the surveyed systems involved patients during design and implementation, from measure choice to evaluation. The ePRO Collaborate system used a codesign process to refine the dashboard and the constructs being assessed in their demonstration trial, combining the use of healthcare coproduction frameworks with stakeholder engagement. 40 Developers of the OPTimising engagement in routine collection of Electronic Patient-Reported Outcomes (OPT-ePRO) system used participatory codesign guided by normalisation process theory to guide design and planning with clinicians, patients and caregivers. 42 Patients with CKD at varying stages were involved in the decision to create a study-specific questionnaire for the RePROM study, in order to capture outcomes they felt were most relevant. 9 Data collection and assessment Table 3 summarises key data collection functions.
Similarly, half the developers reported using a newly developed PRO measure, either completely new or a mix of existing tools. Only three surveyed systems had ePRO availability in languages other than English, 20 21 41 43 although several developers reported plans for future translations, dependent on funding. There were a mixture of mechanisms to remind patients to complete Open access their ePROs, with email (50%) and verbal reminders (50%) being most popular. Reminders were seen as the best way to avoid missing data (55.6%), but some systems offered a save and return function.
Frequency of reporting was varied, with most common timing of assessment being prior to a clinical assessment (40%). Some systems did have set time points ranging from 1 to 6 monthly. Two systems allowed patients to complete as required. 9 41 System reporting Table 4 outlines system-reporting features.
All systems allowed patients and clinicians direct access to ePRO results or summary reports. Doctors (44.4%) and nurses (55.6%) were responsible for initial review and action of results. Follow-up (face to face, virtual, telephone) was largely dependent on scores. Electronic PRO reports contained current scores (summary and individual scores) in 70% of systems, 80% presented longitudinal changes and 40% offered cut scores (low, medium and high). One system was part of an overall dashboard with the option for patients to combine ePRO scores with lab values in a graph, 40 and the Ambuflex PRO questionnaire was also viewed alongside other available clinical data, such as biochemistry, blood pressure, body weight, as a decision aid together to determine whether the patient needs a phone call or a face-to-face visit. 24 In terms of data visualisation, graphical presentation was most popular (80%) with 50% of ePROs using colour coding to highlight significance of scores. Less frequently used approaches of score presentation were tables (30%), numerical scores (30%) and emoticons (10%). Assistance with interpretation within the report was uncommon, but 30% of ePROs had developed a safety alert system dependent on respondent scores. 9 20 25 The RePROM system produced real-time email alerts directly to the kidney nursing team, 9 while the SMaRRT HD system generated email alerts, for which the severity alert thresholds required adaption, so that they were frequent enough to ensure symptom capture without overburdening the workflow with emails (range 2-22/week). 20 The Ambu-Flex system keeps track by giving patients a red, yellow and green status with non-responders presented to clinicians on an alert list. 25 The eNephro study by Thilly et al 44 was investigating the cost-effectiveness of home telemonitoring in patients with CKD. This system included a clinical decision support algorithm to detect situations of risk; relying largely on biological parameters, it could alert in cases of a complication associated with predefined symptoms. 44 Follow-up interviews Semistructured interviews were undertaken with seven ePRO developers, allowing clarification of survey responses: to explore the development process and identify contextual factors working for and against implementation. All interviews took place online using videoconferencing software (Zoom) with no-one else present. Analysis generated rich data which are presented in line with the CFIR key domains. 35 Figure 1 outlines the conceptual framework (Illustrative quotations to support the key findings are provided in online supplemental file 5).

Intervention characteristics
Analysis highlighted the heterogeneity of the systems. However, developers agreed that systems need to be flexible, responsive, and follow an iterative development strategy. All interview participants felt that their systems would need future refinement, whether simple adaption to increase language availability or more complex programming to allow real-time symptom alerting or provision of self-management tools. Knowledge of context (population and setting) and purpose of ePRO data collection should be clearly defined and communicated to all stakeholders including patients. Developers acknowledged that the evidence of ePROs benefit is less mature in nephrology than other clinical specialities. However, participants indicated ePROs could deliver a relative advantage over current care delivery: improving communication, supporting symptom management, enhancing self-care and shared decision making, with potential to reduce healthcare utilisation. However, all participants recognised the challenges, particularly of clinician engagement. They cited clinician anxieties around potential increases in workload, attendance to symptoms and problems unrelated to kidney disease. Consequently, several developers chose not to include real-time safety alerts associated with severe symptom reports.
Several systems used currently available infrastructure, utilising existing platforms, registries, and validated measures. While this offered the advantage of existing expertise and reduced cost, this was offset by restrictions on local adaption and global comparability due to lack of harmonisation.
Outer setting All ePROs system programmes had been impacted and delayed by the COVID-19 pandemic. Several developers reported that key staff including informatics teams had reprioritised activity and had less availability to support ePRO implementation. However, they stated that the rapid shift to virtual consultations and recognition of the potential impact of ePROs in managing care, particularly outpatient planning, invigorated interest in ePROs.
Developers highlighted the importance of partnership and collaboration -across clinical, academic, and commercial organisations. Registries were seen as an ideal collection platform. However, developers indicated multi-organisational approaches could lead to competing interests and governance issues that can cause delays.
Developers acknowledged the value of integration of ePROs directly into the EHR, but not all systems had this functionality. Some still required manual entry, which was reported as time consuming, costly and would not Open access support real-time care. Many systems were standalone in nephrology secondary care, meaning data would not 'follow' patients across care settings and could not be used to manage transition of care needs.

Inner setting
Developers suggested potential unintended consequences of ePRO implementation included further widening of health inequities, due to inadequate accessibility to digital devices, variable digital competence and lack of inclusivity caused by poor availability of systems in multiple languages or failure to assess cultural compatibility. Some developers were investigating such outcomes in particularly vulnerable groups, for example, elderly, ethnic minorities.
Developers cited the continued prevalence of biomedical models of care, despite general acceptance of the need to move to person-centred care, as a barrier to ePRO implementation. They indicated approaches for shifting 'power' to patients by focusing on patient understanding of ePROs and involvement at all stages of design and implementation, along with patient peer support to counteract cultural barriers.
Doctors were the professional group most often involved in review and action of ePRO responses, but several systems directed ePRO scores to nursing staff or physician assistants as the initial handler. Resources needed to be available to allow the engagement of the wider multidisciplinary team, including supportive and rehabilitation services.
While clinicians were largely receptive to ePROs, workflow compatibility was seen as crucial to uptake, and this was reflected in the preference for a focus on symptom management as primary objective of capture. Several systems included free text options to report symptoms or problems and clinicians needed reassurance that this would not significantly impact workload. Participants reported that pilot study data suggested free text responses did not adversely increase workload but provided another mechanism for patient/provider communication, likewise with initial data on use of symptom report alerts.

Characteristics of individuals
Participants discussed the importance of measuring outcomes, which matter most to patients but emphasised the hypothetical danger of over digitising the patient experience, meaning that clinicians risk responding to scores alone rather than listening and connecting with the patient to further interpret ePRO data. Several participants looked forward to a future when ePROs might be given similar status to other forms of medical data. They noted clinicians frequently discuss burden associated with the collection and use of ePROs but did not consider similar factors associated with non-essential blood or invasive tests.
Developers reflected on the varied ability to deal effectively with responses, both at an organisational and individual level. They recognised the heavy and complex workload within the specialty, and while some systems offered management guides or further resources, participants were not all in favour of prescriptive treatment decision algorithms, some preferring to allow the exercise of clinical judgement.
The process of implementation Systems were in different stages of development and use, with some being part of wider research or quality improvement programmes. A pragmatic approach to implementation was often described: for example, measures were chosen according to what was already being used and what patients would be prepared to complete. Existing platforms that had been previously assessed for governance and regulatory compliance were exploited. Implementation readiness for use in routine care was affected by most systems being investigated outside a real-world context. Sustainability was an issue: future planning would need to include activities undertaken by research personnel, including administration tasks, completion support and training. Long-term IT and informatics support needs to be factored into costing for maintenance and adaption.
All developers stated the importance of patient involvement at all stages and outlined different ways patients had offered input and insight into codesign/production to research participation in usability testing. Staff dedicated to engagement were useful, ensuring extended community groups were involved.
Education and training associated with the knowledge needed to support ePROs appeared to be underdeveloped when compared with other aspects of system design. Training resources were often in development and current iterations focused predominantly on system use and did not extend to interpreting or actioning responses. It was recognised that training was time-consuming and needed to be ongoing.
Overarching themes: barriers and facilitators to effective implementation Across all CFIR constructs, analysis identified potential barriers and facilitators. Barriers included the culture associated with biomedical models, variable clinical engagement particularly from doctors and governance issues. Participants suggested facilitators included the development of patient-driven systems, utilisation of clinical champions, who need both time and belief to promote the system, and adapting existing resources.

DISCUSSION
This review summarises and integrates survey and interview data on the features and implementation factors of 15 nephrology ePRO systems used across seven countries with table 5, outlining key priorities for successful implementation.
The majority of surveyed systems prioritised symptom management over other uses of ePROs such as benchmarking and commissioning (see table 2). While some Open access systems were designed to collect ePROs across CKD stages and treatment modalities, patients undergoing HD were the most common population for ePROs use. According to recent reviews, patients with CKD report between 56 and 68 common and/or severe signs and symptoms, with the exact symptom burden often determined by stage of disease and treatment. 3 45 Participants in this study reflected that symptom management was a good primary objective on which to concentrate early implementation efforts, with capability to add additional functionality and use.
None of the surveyed systems was reported as being developed to align with national quality measures, such as two times yearly In-Center Hemodialysis Consumer Assessment of Healthcare Providers and Systems 46 and annual assessment of HRQOL using the KDQOL-36 included in the U.S. Medicare End Stage Renal Disease Quality Incentive Program. 47 However, increased utilisation of national registries to collect ePROs for routine care and population level use 48 highlights the potential to assess HRQoL and symptoms through a quality registry.

Stakeholder involvement
This study highlights the increased development of nephrology ePROs in the last 5 years, with all developers reporting positive and negative effects of the COVID-19 pandemic. While there were delays to implementation due to reprioritisation of activity, the increased drive to deliver patient centred care via telemedicine has enhanced attention on ePROs.
Patient and public involvement (PPI) is a prerequisite to inclusive and equitable PROs' research 49 and the meaningful involvement of patients in ePROs implementation programmes is crucial to effective uptake. 50 Wellestablished PPI groups, reflective of the total population, can offer advice on measure choice, language requirements and support resources. Patient input in design of software features including reporting and data visualisation can ensure data is interpreted correctly 51 using various devices such as graphs, emoticons and heat maps. 52 53 Patient 'navigators' can offer health literacy and communication peer support for successful digital completion. 54 All the surveyed systems had involved patients in aspects of development from measure choice to evaluation. Involve all stakeholders-define and communicate key system objectives Use flexible system design to facilitate local, regional and national compatibility Support localised adaption that is, clinic level and resourcing EHR compatibility Allow varying modes of administration, including paper Reminders-varying forms (email, app alert, etc.) Alert systems-develop pathways and algorithms for action that compliment workflow High levels of automation to reduce workload Data collection features-need efficient, easy user experience Easy to access, save, submit and review Optimise data interpretation and visualisation Infrastructure support to deliver 'holistic care'-treatment and care algorithms Appointment and support for clinical 'champions' Patient peer support via 'Patient Navigators' Training packages to support implementations for all users Sustained IT support Ongoing evaluation programmes Localised education on the management of long-term conditions Micro level priorities (individual) involve all key stakeholders and undertake meaningful PPI activity to ensure ePROs are accessible and inclusive for population Identify local needs that is, language availability requirements Provide support for those with poor health or digital literacy (need an assessment of needs) Consider multi-media PROMs Optimise accessibility-allow onsite completion via kiosk or tablet, enable 'bring your own device' Evaluation by patients to ensure systems are not over-digitising the patient experience by not discussing responses with individuals Meaningful follow-up of PROs responses, tailored to individual needs

Open access
While other important contributors to ePROs implementation are quality assurance, governance, informatics and information technology departments, any successful ePRO system will need effective clinician 'buy in'. This study identified this as a key challenge, particularly from those clinicians not involved in ePROs research. Clinical staff state they are anxious about disruptions to already tight workflows and while ePROs are seen as a useful conversation starter, clinicians fear ePROs lack clinical utility for decision-making, citing the current lack of empirical evidence on the benefits of using this data in nephrology. 32 55 Indicating a need for research to improve the evidence base of ePROs use at individual and aggregated levels, 56 including studies which employ cluster designs and use techniques to maintain allocation concealment. 57 Studies included in this review are going some way to provide this evidence.
Developers described utilising the assistance of 'clinician champions'; individuals with both the time and influence to drive ePRO implementation. These champions would encourage clinicians to view ePRO scores as key health data, in the same way, as a biomarker or a lab test, to manage care.
Studies have highlighted tension from clinicians in their ability to respond to ePROs, whether due to workload or belief that their scope of practice should be confined to matters related to nephrology. 32 58 59 Some clinicians indicate that they are more comfortable assessing, rather than actively managing, psychosocial symptoms. 41 While holistic care is advocated within kidney care, it is not necessarily supported within the system, and multidisciplinary approaches and further training in managing mental health problems are needed. 60 Although research from other specialities suggests patient encounters using ePROs do not take longer, 61 studies on workflow changes and impact are required. Training in all systems was predominantly linked to ePROs system functionality; formal interpretation guidance beyond provision of cut scores or basic colour coding was rare or 'to be developed'. Electronic questionnaires which use and calculate scores or colour codes specifically for decision aids for treatment are considered medical devices and must comply with relevant legislation. 62 Most systems offered or were developing self-management guidance for patients. Effective training need analysis at the planning stage is crucial to assess the support needed by the multidisciplinary team to respond to ePROs. In general, systems were being developed iteratively, with training and support materials being developed or expanded after launch and this may be counterproductive to early and sustained engagement from clinical teams.
Further research is needed to investigate how we interpret PROs for CKD management and event prediction, such as dialysis start time, the impact of changing dialysis modality or the decision to undertake a conservative care pathway.
Real-world implementation A shift to implementation science for guidance on how to promote adoption, enhance readiness and optimise use of ePROs in real-world settings will also identify any unintended consequences of use, such as exacerbation of health inequalities and potential over digitisation of the patient experience. 63 64 Many groups who are already subjected to disadvantage and worse health outcomes are also subjected to digital exclusion. 65 Flexible systems enhance accessibility, offering patients' choice on time, frequency, mode of administration and place of completion. Electronic PROs are commonly completed in a healthcare facility via tablet, where completion support is available. However, onsite completion may impact workflow and support assistance traditionally delivered by research staff will need to be factored into sustainability plans. Home completion may need to be incentivised and alternative means of completion should be offered as required, that is, voice activation response systems, hardcopy by post, following assessment of local population needs. Combinations of digital inclusion approaches are likely needed, to support patients with the digital and health literacy skills required to negotiate both their EHR and any corresponding ePRO data 66 and offer access to hardware. 49 65 Early findings from the EMPATHY study stress the difficulty in implementing ePROs in HD settings, indicating that routine PROs use failed to demonstrate a significant improvement in patient-clinician communication. 67 While these findings suggested relatively good communication pretrial, the qualitative data offered potential reasons for no effect: insufficient patient and clinician understanding of the purpose of PROs, challenges with administration, inconsistencies with PROs as communication tools and limited perceived value. Highlighting that evaluation of current ePROs systems should be investigating implementation outcomes (ie, fidelity, appropriateness, acceptability, feasibility, reach, adoption and sustainability) 20 40 as well as whether ePROs are excluding already disadvantaged groups. 68 Evaluation using implementation science methodology, while formally investigating the effectiveness of ePROs interventions, offers a potential avenue to build the evidence base on benefit while supporting adoption, adaption and sustainability. 69 While effort was made to comprehensively identify nephrology ePRO systems for inclusion, a key methodological limitation was the possibility that systems which were not referenced in the literature, available in English, would be missed. This method placed reliance on appropriate and reliable indexing systems. However, consultation with experts in the field, including study participants, to identify suitable systems aimed to minimise these effects. It is suggested that as ePROs systems proliferate in healthcare settings, an international prospective register of clinical ePROs systems could offer one solution for knowledge transfer and collaboration, but issues of ownership, resource and upkeep would need to be addressed. Additionally, while the sample sizes were small, sampling Open access was purposive, with cases being selected by virtue of their capacity to provide richly textured information, relevant to the phenomenon under investigation. 70 The prevailing concept for sample size in qualitative research is 'saturation'; however, this study has been guided by the concept of 'information power', that is, the larger information power the sample holds, the lower N is needed. 71 The specificity of experiences and knowledge offered by the participants for an in-depth analysis meant the actual sample held adequate information power to develop new knowledge, referring to the aim of this study.
Areas for future research Data linkage is central to multiple usages and harmonisation of collection platforms and associated measures will support this. Integration of ePROs within the EHR can offer 'one stop' access for clinicians and patients, if given access. Extended accessibility across healthcare providers would allow long-term monitoring across the kidney care continuum. A study of daily ePROs collection reported specific events such as fistula formation and modality changes led to PRO changes, demonstrating PROs can capture differential patient experience in CKD. 72 This study highlights the need to both implement measures that are sensitive and responsive enough to detect clinically relevant change, such as disease progression, over both short and long term, while considering how best to minimise respondent burden.
New technologies offer a potential solution, computerised adaptive tests (CATs) offer sophisticated measure delivery using algorithms based on item response theory to personalise ePROs for patients. CAT-based measures are shorter, more accurate and efficient. 73 None of the reviewed systems was currently using CAT administration; however, several developers reported that this was where they saw the future of ePROs and were actively researching this area; a renal-specific item bank and associated CAT is currently being developed for use in the UK. 74 To conclude, while there has been considerable research investment in the development of ePROs, to measure CKD symptoms and HRQOL, the next step is to accelerate the implementation gap between research and practice; this study supports this objective by outlining key system features and exploring factors to optimise the delivery ePROs in routine care settings within nephrology. Contributors NEA, MC, CM, PC, OLA and DK were involved in the conception of the study, and methodological design. NEA conducted data acquisition. NEA and CM undertook analysis and interpretation. NEA wrote the first and revised drafts of the manuscript. RV is a Patient Partner, involved in study design and review of key study documents including protocol, participant information sheet, survey and topic guide. All authors were involved in the final approval of the version to be submitted and published, and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work were appropriately investigated and resolved. NEA is responsible for the overall content as the guarantor. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care. The study sponsor and funders have no role in study design, including collection, management, analysis, and interpretation of data; writing of the report and the decision to submit the report for publication.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Study design section for further details.
Patient consent for publication Not applicable.
Ethics approval This study involves human participants and was approved by Interviews with professional experts-Ethics approval was issued on 06/07/2021