Article Text

Download PDFPDF

Original research
Piloting a web-based systematic collection and reporting of patient-reported outcome measures and patient-reported experience measures in chronic heart failure
  1. Francesca Pennucci1,
  2. Sabina De Rosis1,
  3. Claudio Passino2
  1. 1Institute of Management - Laboratorio Management e Sanità, Scuola Superiore Sant'Anna, Pisa, PI, Italy
  2. 2UOC Cardiologia e Medicina Cardiovascolare, Fondazione Toscana Gabriele Monasterio per la Ricerca Medica e di Sanità Pubblica, Pisa, Italy
  1. Correspondence to Dr Francesca Pennucci; f.pennucci{at}


Objectives To evaluate the feasibility of a digital and continuous collection and reporting of patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) for chronic heart failure (CHF).

Design A single-site pilot study was settled for evaluating the feasibility of the intervention, both using qualitative and quantitative data (ie, workshop, surveys).

Setting The pilot has been implemented in a Tuscan specialised hospital (Italy).

Participants 162 patients were involved. Inclusion criteria were: a previous diagnosis of HF, age ≥18 years, absence of cognitive impairment or active tumours, ability to provide informed consent to study participation.

Intervention The continuous collection and reporting of PROMs and PREMs has been designed and implemented in 2018. PREMs questionnaires for patients were developed, while Kansas City Cardiomyopathy Questionnaire-12 was used for assessing PROMs. Questionnaires are administered at specific time points: discharge; 30 days, 7 and 12 months after the discharge. Enrolment of patients, administration and real-time reporting of questionnaires are carried on through a digital platform.

Outcome measures Enrolment, response and drop-out rates were considered to assess the feasibility of the intervention. Qualitative data were collected during meetings and workshops with health workers. The representativeness of the recruited sample with respect to the population characteristics was also evaluated.

Results The system has been successfully implemented during 2018. Response rates have been consistently above 50%, demonstrating patients’ transversal willingness to participate. All the involved stakeholders acknowledged the feasibility of the design. The recruited sample is significantly different in terms of age and educational level compared with the overall population characteristics.

Conclusion It is possible to run a web-based systematic collection and reporting system for CHF patient-reported data. Systematic collection and reporting of PROMs and PREMs data allows professionals to increasingly assume CHF patient perspective in their daily work. Limitations will be used to improve the system.

  • patient-reported measures
  • chronic heart failure
  • healthcare management
  • value-based
  • performance evaluation

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • This is the first pilot study of a continuous and systematic data collection of patient-reported measures related to chronic or long-term care field.

  • The study design can be applied for testing the feasibility of a similar system developed for other chronic diseases and in other geographical and cultural contexts.

  • Digital platform to manage data collection and reporting, professionals’ engagement, longitudinal assessment of both patient-reported outcome measures and patient-reported experience measures emerged as facilitating factors to implementing the intervention.

  • Professionals’ role in enrolling patients is demanding and can produce selection biases in the sample.

  • The use of digital tools can produce self-selection biases due to digital divide.


In the healthcare sector, public organisations (POs) are in charge of providing high quality healthcare services, guaranteeing equity and sustainability of the system itself. Healthcare organisations should produce value for their users (personal value), as well as for citizens in general (social value). In order to monitor value creation processes, administrative and clinical data can be complemented by patient-reported measures. There is no evidence on successful experiences of continuous and systematic data collection of patient-reported measures related to chronic or long-term care field. This study presents a pilot experience of implementing a specific design for assessing chronic heart failure (CHF) patients’ pathway, which can be easily replicated in other geographical and cultural contexts.


Over the last 20 years, the definition of value in healthcare has changed.1 2 Analysing the input–output balance is not anymore considered sufficient to measure value, which is now intended as broader results/benefits to be obtained at the individual level.3 4 The focus has moved to outcomes produced for the individual patient,3 and to the whole population.5 For healthcare systems with universal coverage, where policy-makers and healthcare managers are responsible for the health of an entire population, it is imperative to follow this model. The value-based healthcare framework has been encompassed by the Triple Value one, which includes patient perspective together with population and system perspectives.6 In this model, value is measured also considering whether resources are used for individuals who would benefit most from their investment. When trying to measure the value, it is necessary to consider the broader impact of a specific care decision or a healthcare policy. In addition, value is measured also as outcomes that matter to the individual patient compared with the cost of achieving the same outcomes. In the Triple Value framework, value produced for the patient is meant as both the technical value, given by the cost–benefit ratio, and the personal value, given by the patient evaluation. Consequently, POs in healthcare have increasingly realised the importance of implementing a patient-centred approach also in measuring healthcare activities, by adopting the patient point of view and following his/her cross-setting care pathway.7–9

Therefore, measuring value in healthcare requires a multidimensional and multiperspective approach, which considers the complexity of healthcare systems in terms of involved actors and competing interests and values.1 3 6 10 Several measurement tools are in place to monitor private and POs’ activities in terms of efficiency and effectiveness.3 11 12 Analysing quantitative and qualitative indicators is fundamental to ensure that inputs generate adequate outputs and outcomes.4 For POs, as stated, this is especially true given that they manage collective resources and should produce both individual and collective benefits guaranteeing financial sustainability and equity.1

Traditional measurement tools, when integrated into a performance evaluation system (PES) and used with proper managerial levers, allowed healthcare POs to improve their performance.13 Nonetheless, indicators from administrative data are still the main sources of information for monitoring and assessing healthcare. Despite the value for patients is often revealed over time and across settings,3 10 measurement systems are still not equipped to longitudinally analyse care pathways and outcomes in terms of quality or integration.1 8 9 The focus is still on specific settings and the organisation perspective.3 14 15 In fact, traditional indicators have been focused on measuring adverse outcomes, such as mortality, infections or readmission rates while many other relevant outcomes (eg, quality of life, social limitations, psychological health, etc) cannot be measured without asking patients’ direct feedback.9 14 15 Over the last decades, researchers have worked to stimulate the inclusion of the patients’ voice inside traditional measurement systems.16 17 Still, what matters to patients and families from their actual point of view is often underestimated and not measured.3 15 18

In this perspective, many countries have started to survey patients in order to add patient-reported experience measures (PREMs) and patient-reported outcome measures (PROMs) to their PES as complementary sources of information besides indicators from administrative data.19–22 In particular, PREMs have been first developed to collect patients’ feedback on experience with healthcare services and used as a powerful tool for improving quality of care.16 PROMs, on the other side, are longitudinally collected to measure effectiveness within clinical trials or for improving individual patients’ health status by means of his or her reported functional, psychological and social outcomes.23

These surveys are generally oriented to measure surgical pathways or punctual experiences of patients, like as the hospitalisation, while longitudinal or outcome measurement for chronic diseases is still not well developed.15 A longitudinal assessment of the care that patients receive and the outcome it produces can be crucial to measure value produced in chronic care pathways, although their complexity and diverse characteristics.3 15 Chronic diseases need a daily management that patients and families mostly do. Thus, gathering their point of view over time is crucial, as well as giving them a central role and responsibility in their care pathways.10 24

There are some examples of establishing measurement of outcomes for long term and chronic patients,11 25 mostly as pilot, experimental and one-time projects.26 Evidence is available on successful experiences of continuous and systematic data collection of patient-reported measures, but they are not related to chronic or long-term care field and still present some issues and barriers yet to be addressed and overcome.19 26 In this respect, the administration method is one of the aspects to be considered when designing data collection systems since, for example, chronic and long-term patients can potentially participate in surveys along their whole life.27 Data reporting tools and methods are also fundamental since collecting measurement of outcomes and experiences helps to generate value if patient-reported data are actually used.23 28

In this paper, the authors will present a pilot experience that was conducted to test the feasibility of implementing a digital, systematic and continuous initiative of patient-reported data collection and reporting for patients with chronic heart failure (CHF). This experience is part of the Tuscan PROMs and PREMs Observatory that will be briefly described in the next paragraphs (further information can be find in refs. 29 30). Specific characteristics of the pilot, its preliminary results and managerial implications will be described and discussed. Future perspectives for improving the model are presented in the end.


Data collection system

In 2018, the Fondazione Toscana Gabriele Monasterio (FTGM), a tertiary referral centre for CHF based in Pisa, Italy, started to collect PROMs and PREMs data from patients hospitalised for CHF. All the patients were invited to participate according to the following inclusion criteria: a previous diagnosis of CHF based on guideline criteria,31 32 age ≥18 years, absence of cognitive impairment or active tumours, and ability to provide informed consent to study participation.

All the eligible patients were enrolled consecutively before discharge from the index episode (hospitalisation), when contact information of study participants and their caregivers was collected (figure 1).

Figure 1

Scheme of enrolment and data collection processes. PREMs, patient-reported experience measures; PROMs, patient-reported outcome measures.

The day after enrolment, a personal link was automatically sent to patients or caregivers by phone or email, according to patient’s preference. The link gave access to the first online questionnaire. Follow-up questionnaires were sent after 1 month, then 7 and 12 months after index hospitalisation. At each time point, patients or caregivers received three reminders to fill the questionnaire (figure 2).

Figure 2

Timeline for administering the questionnaires with a synthetic list of measures. PREMs, patient-reported experience measures; PROMs, patient-reported outcome measures; KCCQ-12, Kansas City Cardiomyopathy Questionnaire-12; SCHFI, Self-Care Heart Failure Index.

Professionals and researchers worked together in selecting or designing questionnaires that include measures of outcomes, experience of care and patient self-care.

Patient-reported outcome measures

The Italian short version of the Kansas City Cardiomyopathy Questionnaire-12 (KCCQ-12) was selected to measure disease-specific outcomes.33 The KCCQ-12 scale explores different dimensions related to CHF: physical limitation (three items), symptoms frequency (four items), social limitation (three items) and quality of life (two items). The total score goes from 0 (worst possible condition) to 100 (best possible condition). The KCCQ-12 was not included in the baseline questionnaire since the recent hospitalisation event would deeply influence patient report of his/her health status, in agreement with International Consortium for Health Outcomes Measurement guidelines.27 By contrast, the KCCQ-12 questionnaire was administered at each follow-up time point, regardless of the occurrence and timing of further HF hospitalisations. At each follow-up time point, a specific question is posed to detect if further hospitalisations actually occurred.

A question on perceived health status in the previous week was included at each time point, including baseline evaluation. Again at each time point, the Italian translation of the Self-Care Heart Failure Index (SCHFI) was included,34 assessing: self-care maintenance (ten items considering all those behaviours that monitor signs and symptoms and maintain HF stable), self-care management (in the subset of patients with dyspnoea or oedema during the previous month: six items measuring symptoms recognition and responses to signs and symptoms of an exacerbation) and self-care confidence (six items assessing patient self-efficacy in performing the entire self-care process). One item of the self-care maintenance domain was modified with the consent of SCHFI authors. This item specifically monitors adherence rate to influenza vaccination, which should be done once a year. This change does not influence the number of items or the algorithm to compute scores.

Patient-reported experience measures

At baseline, patients were asked questions exploring the quality of care before the index hospitalisation and during the hospital stay, distributed into nine items with Likert, single or multiple choice responses (the baseline questionnaire is included in online supplemental appendix A): health professionals caring for them (one item), access to the hospital (one item), previous admissions for HF (three items), coordination between general practitioners (GPs) and cardiologists before the hospitalisation (one item), pharmaceutical dimension (three items).

After 1 month, the questions were related to the experience of care during the index hospitalisation (GP’s role, two items; length of stay, one item; patient and family involvement, two items; emotional support, two items; teamwork of staff, one item; comfort, four items; overall satisfaction, one item), discharge management and organisation of home care (communication efficacy and clarity, six items; follow-up visits, two items; home care, one item; medical aids, one item; out-of-pocket expenditure, one item).

Seven and 12 months after the baseline hospitalisation, questions explored monitoring by clinicians (one item), coordination of care during follow-up (one item), home care (two items), out-of-pocket expenditure (one item), occurrence of acute events (five items), pharmaceutical dimension (four items), follow-up visits (two items).

Data reporting

Data from PROMs and PREMs were collected on a dedicated online platform. Data were managed in an aggregated and anonymised fashion. FTGM specialists involved in the project could access the online platform at any time to check enrolment and aggregated results. The aim of this daily updated platform is to allow managers and professionals having ready-to-use information in order to monitor their activities in terms of patients' experience and outcomes. A thorough data analysis was planned once a year. The present study reports data from the first yearly analysis.

Statistical analysis

Statistical analysis was performed using the STATA software (V.15). The Coarsened Exact Matching procedure was applied35 to match the observations from the PROMs and PREMs survey to the observations from the electronic health records (EHRs), thus identifying the more similar records between the two datasets. Diagnosis of CHF has been used to select patients inside the EHRs. Afterward, age, sex and educational level were considered as variables for matching the records between the two datasets. The matched dataset has been used to verifying if the characteristics of all the discharged FTGM patients recorded in the EHRs and the ones who replied to the first PROMs and PREMs questionnaire are significantly different. The same analysis was performed only considering the records of the PROMs and PREMs dataset, through the comparison of respondents and non-respondents’ characteristics.

Statistical tests on mean values (t-test) or on bivariate distributions (χ2 test) were performed. More in details (figure 3):

Figure 3

Datasets and population for the comparison of characteristics. EHR, electronic health records; FTGM, Fondazione Toscana Gabriele Monasterio; PREMs, patient-reported experience measures; PROMs, patient-reported outcome measures.

  • Patients recorded in the EHRs and patients who replied to the PROMs and PREMs survey were compared on age, sex and education level.

  • Respondents and non-respondents to the PROMs and PREMs survey were compared on age, time since diagnosis, ejection fraction, NYHA class, aetiology, number of comorbidities.

Patient and public involvement

Patients were involved in data collection and in giving feedbacks about the functioning of the platform. The intervention has been designed keeping in mind their specific care pathway, which services they experience during it, which are the non-clinical domains impacted by the pathway.


From February 2018 to February 2019, 162 patients were enrolled for the PROMs and PREMs survey, corresponding to 27% of patients admitted for worsening HF during the same period. Among them, 104 (64%) completed the questionnaire at baseline, 95 (61%) at 1 month, 41 (49%) at 7 months and 9 (31%) at 12 months.

Around 30% of the patients gave only a caregiver contact while the majority of participants gave a personal contact (70%). Among these latter, 39% gave both their telephone number and their email address while only 14 of them gave only the email address.

Patients giving their email contact tend to reply more often than other patients do (t0 response rate 67% vs 64.2%; p=0.09). Response rate was significantly higher when patients gave only a caregiver contact (80% vs 64.2%; p=0.02), and became even higher if this contact information was an email address (84% vs 80%; p=0.005).

Patients and caregivers demonstrated their engagement filling in the questionnaires and contacting research staff regularly in order to pose questions about the survey and signal platform bugs or relevant details regarding their specific care pathway.

Table 1 reports descriptive statistics comparing patients who replied to the first PROMs and PREMs questionnaire versus CHF patients discharged during the same year and registered in the EHRs dataset. PROMs and PREMs sample presents a greater percentage of males compared with CHF FTGM patients in EHRs (p<0.04). There is also a statistically significant difference on education: PROMs and PREMs respondents are more educated (p<0.02).

Table 1

Descriptive statistics for respondents CHF patients to PROMs and PREMs survey and CHF patients discharged during the same year in FTGM

A second comparison to check for self-selection bias was made between respondent and non-respondents to the first PROMs and PREMs questionnaire. No statistically significant differences emerged (table 2).

Table 2

Descriptive statistics for respondents and non-respondents to PROMs and PREMs survey

Some other characteristics of PROMs and PREMs respondents are presented in table 3. The majority of them were retired (67%), while the 12% had a stable job. Around 10% of PROMs respondents lived alone and around 67% of them declared to be married.

Table 3

Descriptive statistics for actual participants into PROMs surveys

Table 4 reports the average scores of PROMs for the cohort of patients who answered to the baseline, 1 and 7 months questionnaires. On average, all the measures show an incremental improvement during time.

Table 4

PROMs scores (patients who replied to the 7 months questionnaire, non-adjusted)

Researchers with all the involved staff have conducted regular meetings to analyse implementation difficulties. Based on the collected feedback, the regular process of enrolment, which was thought as completely managed by professionals, has been reshaped involving also nurses to illustrate the project and collect the information needed to register patients into the online platform.

Two formal workshops have been organised to present and discuss the collected data.

  • The first workshop was aimed at critically analysing the feasibility of implementing and managing PROMs and PREMs data collection after the first year of activity. Looking at the preliminary analyses of data and sharing which were the facilitating factors and the barriers, healthcare workers involved in the project could reflect on what could be valued or adjusted in the following year of activity.

  • In the second workshop, data were presented and discussed with cardiologists working in the same Local Health Authority (LHA) of FTGM thus serving the same population. The aim of this meeting was to explore the possibility of expanding the data collection to the whole geographic area so to have patient-reported measures from the other hospitals to be compared. They demonstrated to be interested and willing to experimenting the implementation of the system tested in FTGM. Some of these discussants have started to enrol patients in their hospitals since January 2019.

Currently, collaborative efforts are in place both to improve the digital platform and to valuing the information patients are giving back. In fact, thanks to health professionals’ feedback a new ameliorated version of the platform is currently available.

A report has been produced to describe the design and implementation processes together with preliminary results of the pilot phase36 and researchers are working to expand data collection also in other Tuscan LHAs and in other Italian and foreign regions, like they have already done with PREMs survey.29 37


A strong interest is emerging about testing tools to collect information on outcomes and experience reported by chronic patients. Both UK and Canada healthcare systems are hypothesising to design similar tools and the Organisation for Economic Co-operation and Development (OECD) Patient-reported Indicator Surveys (PaRIS) initiative is currently in its design phase.11 15 25

The presented Tuscan experience shows that it is possible to implement a systematic and continuous collection and real-time reporting of PROMs and PREMs data for CHF patients within a public funded health system by means of a digital platform.

In particular, the Tuscan experience has some unique characteristics compared with other PROMs and PREMs collection experiences in respect of:

  • Survey administration: It is structured to collect longitudinal data administering four questionnaires per each enrolled patient.

  • Data use: Patients’ answers are collected and real time reported right after the experience. Data are longitudinally reported on an online platform allowing for daily, weekly, monthly use both for improving organisational processes and for monitoring hospital staff work and interprofessional coordination. In Tuscany, the permanent PROMs and PREMs observatory is included into the already active PES and the collected data will be used also in benchmarking LHAs on their performance.

  • Included measures: The presented experience includes questionnaires that measure generic and specific PROMs together with experience of care and self-care dimensions.

Regarding this last point, it was found that patients’ perception of outcomes could affect the experience rating and vice versa.38 This suggests that PROMs and PREMs should be collected in an integrated way, to understand the relationship between them along the patient care pathway. To the best of our knowledge, there are not ongoing continuous, systematic, experiences of integrated collection of PROMs and PREMs.

FTGM experience has been demonstrated to be easily usable and scalable. Working with professionals and managers, it is possible to match the implementation and routine adoption of PROMs and PREMs survey with their current practice of works. Having real-time available data improves physicians’ usage of them.39 Furthermore, the digital platform requires a limited effort from physicians with data collection, analysis and reporting that are completely managed externally. New enrollers can be included whenever it becomes necessary and this characteristic allowed to engaging new hospitals and patients in a relative short period. This easiness ensures usability for both patients and professionals.14

The longitudinal approach is fundamental to reveal if and how much value is created for patients.3 10 The items measuring integration of care and quality of life can give indications on how to manage processes and practices to improve outcomes.

All the interested actors should be involved to overcome the existing barriers to the extension and use of the tool. In particular, scholars are stressing that health professional should be properly informed that a little effort from their side could produce deep advantages in terms of care quality, patient satisfaction and outcomes.19 40 All the meetings and workshops conducted during FTGM pilot experience have been fundamental moments to consolidate healthcare workers’ involvement and to build their trust in patient reporting and data usefulness. Showing administrative and patient-reported data together made them reflect on possible implications on both processes and outcomes of care.

In the beginning, health professionals were sceptic about the feasibility of involving old and sick patients to participate in a web-based and longitudinal collection design. Adhesion and response rates have proved that patients are willing to actively and consistently participate over time, in accordance with Wagle observation on their capacity of using digital tools.41

Some significant differences emerge in terms of sample selection comparing the FTGM CHF records in EHRs and the characteristics of patients enrolled in the PROMs and PREMs survey. Considering the current model of patient enrolment, it is possible to identify at least two sources of bias: first, patients’ self-selection in terms of who consent to participate; second, clinicians’ enrolment activity that is based on personal evaluation of patients’ capacity in participating to the survey. The first source of bias is always present when dealing with surveys, but a possible solution to clinicians’ discretion in inviting patients is to include directly into the EHR system an algorithm to automatically enrol patients. Professionals would have a lower burden and the enrolment would not depend directly on their choices. However, there could be a side effect in terms of professionals’ involvement inside the project and therefore a risk of lowering patient participation and data usage.

A difference on educational level also emerged. One possible explanation is that highly educated patients are facilitated in accomplishing the task of accessing and filling in the questionnaires considered their knowledge and greater amount of cognitive and material resources.

On the other side, no statistically significant difference emerged when comparing between respondent and non-respondent CHF patients to the PROMs and PREMs survey. This evidence can be a first indication that once a patient consent to participate, then she/he will fill in at least the baseline questionnaire. A further check should be done on the phenomenon called attrition, which refers to selection biases occurring from time to time in association to answers to the following questionnaires during the year of participation.42

The vast majority of PROMs and PREMs participants do not live alone and a great part of them are married. This element can be considered as a proxy of available support when needed, thus the high participation into the survey could be linked to this aspect. The Italian context is changing, but it still characterised from family and social support networks that are strongly involved into disease management, especially when this latter is chronically present.43

Finally, measuring broader outcomes can enhance accountability of health systems increasing the level of responsibility for each involved actor in care delivery activities.40 The inclusion of PROMs and PREMs data inside the Tuscan PES will improve the evaluation of CHF management practices. In fact, a coordinated and multiperspective vision of care in CHF patients’ pathway can give information on both processes and outcomes dimensions.

Practice implications

A pilot experience of implementing a digital, automatic and continuous platform to collect and report PROMs and PREMs data for patients with CHF has been presented in this paper. The implementation succeeded and the digital data collection is currently ongoing and involves other two hospitals in Tuscany.

The study design here presented can be replicated in other geographical and cultural settings, as well as in other chronic pathways. The presented results can be useful to support other healthcare organisations in implementing a PROMs and PREMs collection in chronic care pathways.

Professionals are involved in collecting, viewing and using patient-reported data in order to discuss and act with quality improvement actions. PROMs and PREMs data are also pushing them in increasingly assuming patient perspective in their daily work.

These data can enrich measurement systems and management approaches, fostering the orientation of healthcare systems towards a pathway perspective, with a particular consideration for patients and families’ needs and preferences. Adopting this different approach, healthcare services can be innovated leading to an improvement of LHAs performance in terms of both individual and population-based value creation.

Further developments will be oriented at augmenting the tool capacity in capturing the whole patient pathway and extending the methodology to measure other chronic care pathways. Furthermore, both the collection of data and their use should gradually improve, allowing, for example, professionals to access individual data and to have a linkage between PROMs and PREMs data and patients’ EHRs.


The authors wish to thank the Fondazione Toscana Gabriele Monasterio’s health workers and the patients who participated in the survey. The authors thank also PREMs and PROMs research teams at the Laboratorio Management e Sanità. The authors are particularly grateful to Sabina Nuti and Michele Emdin for their vision, supervision and support.



  • Twitter @SabinaDeRosis

  • Contributors FP and SDR designed the paper and collected data. FP performed the analyses and drafted the manuscript. All the authors contributed to the interpretation of results. CP critically revised the whole work. All the authors gave the final approval of the version tobe published.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval The study conformed with the 1975 Declaration of Helsinki and was approved by the institutional ethics committee.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement No additional data are available.