Article Text
PDF
Abstract
Objective To determine the role of patient demographics, care domains and self-perceived health status in the analysis and interpretation of results from the Canadian Patient Experience Survey–Inpatient Care.
Design Cross-sectional survey.
Setting Single large Canadian two campus tertiary care academic centre.
Participants Random sampling of hospital patients postdischarge.
Intervention and main outcome measures Logistic regression models were developed to analyse topbox scoring on four questions of global care (rate experience, recommend hospital, rate hospital, overall helped). Means of each composite domain were correlated to the four overall scores at the patient level to determine Spearman’s rank correlation coefficients which were plotted against the overall (hospital) domain score for the key driver analysis.
Results Topbox scoring was decreased with worse degrees of perceived physical and mental health in all four global questions (p<0.05). Female gender and higher levels of education were associated with worse scoring on rate experience, recommend hospital and rate hospital (p<0.001). Whereas there was a significant difference between hospital departments in unadjusted measures, these differences were no longer evident after adjustment with patient covariates. Key driver analysis identified person-centred care, care transition and the domain related to emergency admission as areas of highest potential for improvement.
Conclusions Global measures of overall care are influenced by patient-perceived physical and mental health. Caution should be exercised in using patient-satisfaction surveys to compare performance between different healthcare provision entities, as apparent differences could be explained by variation in patient mix rather than variation in performance.
- patient experience
- organisation of health services
- patient experience
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study involves the novel linkage of a clinical database to individual survey results to allow the accurate analysis of the role of patient characteristics and demographics on survey response.
The study provides a validated process by which covariates could be used to adjust patient experience survey outcomes to facilitate interunit and interinstitution comparisons.
The analysis has been completed on data from a single institution and thus the generalisability is not known.
This study is limited by survey non-responders and the random nature of survey participants among the total discharge population from the hospital.
Introduction
Patient experience is now recognised as a critical component of modern healthcare delivery.1 Aside from the clear rationale to routinely provide compassionate care, there exists a strong ethical basis for physicians to support excellence in this area as it is of vital interest to patients and governments as a foundation of patient-centred medicine.2 There is also supportive evidence that improved patient experience may positively impact outcomes,1 3 particularly through better compliance to evidence-based guidelines, such as in areas of chronic disease management.4
There are many different processes by which inpatient patient experience has been measured internationally.5–8 In the USA, it is measured using the Hospital Consumer Assessment of Health Care Provider Systems (HCAHPS) survey.9 Hospital funding from Medicare is partially dependent on the results from this survey, and thus, healthcare organisations are deeply committed to improving results. A modification of the HCAHPS survey (Canadian Patient Experience Survey–Inpatient Care, CPES-IC) was developed through collaboration between the Canadian Institute for Health Information (CIHI), Accreditation Canada, the Canadian Patient Safety Institute, the Change Foundation and the Inter-Jurisdictional Patient Experience Group, and this survey is now routinely administered in four provinces in Canada.10
Though the HCAHPS and the CPES-IC are very similar, there are subtle differences that reflect the unique nature of the single-payer system in Canada. The CPES-IC survey consists of 22 questions derived from the HCAHPS as well as other questions that ‘address key areas relevant to the Canadian context’. The questions can be classified in three specific groups. In the first group, individual questions can be clustered as they reflect care in particular domains such as doctor communication skills (three questions) and nursing communication skills (three questions) among others. The Canadian survey includes the same domains as the HCAHPS, but also comprises several questions that constitute new domains not addressed in the HCAHPS survey such as admission experience, person-centred care, discharge and transition. Further details regarding differences between the Canadian and American surveys are available on the CIHI website (https://www.cihi.ca/en/patient-experience).
The composite questions for each domain can be averaged to provide a mean value which is currently reported at the hospital level for the HCAHPS survey.11 In the second group, there are four questions that reflect overall care that are of particular importance at the institutional level to assess the quality of patient experience. One of these questions is also used as a corporate measure of key interest (‘Rate your experience?’), and it is most commonly used to rank hospitals nationally after adjustment for regional differences.3 Results from the three other questions related to overall care include: ‘Would you recommend this hospital to your friends and family?’ (recommend hospital) and ‘Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible, what number would you use to rate this hospital during your stay?’ (rate hospital) and ‘Overall, do you feel you were helped by your hospital stay?) (overall helped). Success in these and other questions are measured by the per cent of ‘topbox’ designation by the patients in which they have ranked a 4 on the recommend hospital question (on a scale of 1–4) or 9 or 10 out of an ordinal scale of 10 for the remaining three questions. The ‘topbox’ metric has been validated and accepted as a marker of excellence in patient experience measurement.12
The final group of questions found in both surveys consists of inquiries regarding patient-perceived health status and demographic topics such as race and education. These questions are referred to a Patient Mix Adjusters (PMA), and they are used in the HCAHPS survey in order to provide risk adjustment, particularly when comparing between geographic regions. The PMA questions for the HCAHPS are reassessed quarterly by the Centers for Medicare and Medicaid Services (CMS) after reviewing national results.
There is limited familiarity in the assessment of patient experience in Canada and the use of such surveys. Although it has been demonstrated that patient sociodemographic factors such as age, ethnicity, sex and socioeconomic class have been shown to influence patient experience responses,13 there is also no understanding of the validity of the PMA questions in adjusting the results of the CPES-IC survey and how they may contribute to credibly compare units or departments within a hospital. In summary, it is not clear how patient factors such as self-described characteristics including perception of mental and physical health, patient demographics and comorbidities impact the results of the Canadian survey on in-hospital patient experience.
The overall objective of this research was to compare the value of the self-described patient characteristics obtained from the survey with covariates obtained from a hospital database, in the development of a statistical model to predict topbox scoring in the four survey questions related to overall care: (a) rate your experience, (b) recommend hospital, (c) rate hospital, (d) overall helped. We also sought to assess how the PMA questions and other data from the hospital database influence patient experience at the hospital and departmental level and to determine how the composite domain measurements influence the four adjusted global measurements.
Methods
This analysis was conducted as a quality assurance project. Data were collected from 1 April 2016 to 30 November 2016 from the CPES-IC Survey (see online supplementary appendix 1) administered by National Research Corporation (NRC; Markham, Ontario). Surveys were distributed in both official languages.
Supplemental material
Patient and public involvement
There was no patient or public involvement in this research in terms of development, design or analysis.
The data were merged with administrative data collected from The Ottawa Hospital Data Warehouse which is a relational database that contains administrative and clinical data for all patients seen at The Ottawa Hospital. Deciles of income class were derived using the Postal Code Conversion File V.6.6 based on data from August 2015 (Statistics Canada). The Elixhauser class was derived using a modification of the Elixhauser comorbidity measure after applying the latter to the hospital data.14 The occurrence of a patient safety indicator event (ie, an in-hospital adverse event) was determined using International Classification of Diseases 10th Revision coding from administrative data.15
The Ottawa Hospital is a large academic tertiary care teaching centre with two inpatient campuses. There are six admitting departments (surgery/ear, nose, and throat (ENT), medicine, obstetrics/gynaecology, family medicine, ophthalmology and psychiatry). A different survey was used in psychiatry and obstetrics, thus these patients were excluded. Ophthalmology was excluded as it is primarily an outpatient service and accounts for less than 1% of admissions. Data from one surgical division (cardiac surgery) and one medical division (cardiology) were not available as administrative data were not linkable to the patient experience data from NRC due to a differing collection and analysis process. Patients who died prior to discharge were excluded from analysis.
Composite domains were identified as follows: communication with doctors (questions 5–7), communication with nurses (questions 1–3), responsiveness of staff (questions 4, 11), communication of medications (questions 16, 17), transition of care (37–39), person-centred care (30–36), direct admission (questions 24, 25) and emergency admission (26–29). The mean was calculated for each patient for each domain as long as more than 50% of the questions in the domain were reported.16 Spearman’s rank correlation coefficients were determined for the continuous value of each domain and the ordinal global question score, and this was plotted against the overall (hospital) domain score for the key driver analysis.17 The median value of the domain scores was used for the vertical separation of the quadrants due to skewness. Points identified in quadrant 1 represent domains with increased potential for improvement due to high correlation with a global score and lower mean value.
Statistical analyses
Patient characteristics across department groups were compared using a χ2 test.
Distribution normality of covariates was tested using the Shapiro-Wilk test.
For categorical variables with equal variances, one-way analysis of variance was used to compare departments, whereas Kruskal-Wallis equality of populations rank test was used for categorical groups with unequal variances.
Multivariable logistic models were developed to test the primary outcomes from the overall care questions ((a) rate experience, (b) recommend hospital, (c) rate hospital, (d) overall helped) reported as dichotomous outcomes representing ‘topbox’ response (9 or 10) or no topbox (<9). The association of each covariate was assessed using likelihood ratio tests by testing the model with and without the variable. Marginal means were determined for each department using the derived model with all of the covariates, as well as with no covariates (unadjusted). In order to compare departments, a Bonferroni correction was used for multiple pairwise comparisons. A p value of <0.05 was considered significant. Analyses were completed using STATA V.14.2.
Results
Patient characteristics
There were 2989 patients who responded to the survey representing hospital admissions under the care of 295 physicians (146 medicine, 110 surgery/ENT, 22 family, 17 obstetrics/gynaecology.). The institution consists of 918 in-hospital beds, geographically situated at two campuses. Characteristics of the patients from the total group and from each department are presented in table 1. There were significant differences between the department groups in terms of physical and mental health, Elixhauser class, admission status, length of stay, age, discharge disposition, marital status and sex.
Characteristics of patients answering patient experience survey
Topbox analysis: overall measures
The results of the multivariable analyses in the derivation of the model for the overall measures (rate experience, recommend hospital, rate hospital and overall helped) are presented in tables 2-5. Decrease in topbox scoring was associated with worse degrees of perceived physical and mental health in all four of the questions. There was a significant relationship with age group in all questions with lowest ORs in patients between the ages of 18 and 34 years. On pairwise comparison, the predicted scores in this group were significantly lower than those in the age groups of 55–64 years and 65–79 years (p<0.05). Increased level of education and female sex were associated with worse scoring in rate experience, recommend hospital and rate hospital questions. Covariates from the institutional database that were significant contributors to the models included discharge disposition to a facility (recommend and rate hospital), marital status (recommend hospital) and ICU stay (rate hospital). Campus site was found to be a factor in rate hospital (p<0.05).
Analysis of covariates associated with topbox designation of the corporate measure of ‘Rate experience’
Analysis of covariates associated with topbox measure of ‘Recommend this hospital’
Analysis of covariates associated with topbox measure of ‘Rate this hospital’
Analysis of covariates associated with topbox measure of ‘Overall helped’
Adjusted and unadjusted department-based predicted measures for rate experience, and recommend hospital are presented in figures 1 and 2. Unadjusted pairwise comparison of rate experience demonstrated a greater likelihood of topbox scoring with surgery as compared with medicine; however, this was not significant (p=0.054). This difference was not seen after adjustment (p=0.911). Unadjusted pairwise comparison of the question rate hospital demonstrated a significant increase in surgery as compared with family medicine; however, this difference was not present in the adjusted model (data not shown). Unadjusted analysis of the overall helped question demonstrated greater likelihood of topbox scoring in surgery as compared with medicine and family medicine, as well as obstetrics and gynaecology as compared with family medicine (p<0.05); however, these comparisons were no longer significant after adjustment for the covariates in the model (data not shown).
Unadjusted and adjusted predicted per cent topbox of corporate indicator ‘Rate your experience’ by hospital department. Error bars represent 95% CI. Difference between surgery/ENT and medicine significant (p=0.05) in unadjusted, however no differences between departments in adjusted. Adjustment was completed using all of the variables in the multivariable model. ENT, ear, nose and throat; Obs/gyn, obstetrics and gynaecology.
Unadjusted and adjusted predicted per cent topbox of corporate indicator ‘Recommend this hospital’ by hospital department. Error bars represent 95% CI. No statistically significant difference between groups. Adjustment was completed using all of the variables in the multivariable model. ENT, ear, nose and throat; Obs/gyn, obstetrics and gynaecology.
Key driver analysis
Key driver analysis of the global question of rate experience is presented in figure 3. Common domains present in quadrant 1 in all four questions include person-centred care, care transition and the domain related to emergency admission processes. Similar patterns were seen with the other three global questions (results not shown).
Key driver analysis: relationship domain composite measures to the global measure of overall experience, direct admission (left) and emergency admission (right). Horizontal black dotted line: mean for all correlation values. Vertical red dotted line: median for all composites. A: communication doctors, B: communication nurses, C: responsiveness staff, D: care transition, E: person-centred care, F: pain management, G: communication medications, H: admission processes emergency, I: admission processes elective. CPES-IC, Canadian Patient Experience Survey–Inpatient Care; emerg, emergency.
Discussion
Results from the CPES-IC survey administered to patients discharged from a large Canadian multicampus health institution were analysed after merging with a comprehensive administrative database. Two patient-answered demographic questions collected from the survey (patient-perceived overall physical and mental health) were significant covariates predicting topbox recognition in all four of the overall care questions. Increasing level of education and female sex were associated with decreased topbox scoring in rate experience, recommend hospital and rate hospital. Discharge to a non-home environment was associated with lower topbox scoring on recommend and rate hospital. The only significant contributors to the models from the hospital database included marital status (recommend hospital) and ICU stay (rate hospital). Economic status, in-hospital adverse events and Elixhauser comorbidity class did not significantly contribute to the models for the four questions related to overall care. After adjustment, there was no significant difference in the predicted measures between the four major departments in any of the four questions that related to the overall patient experience. Finally, key driver analysis using these models confirmed that the greatest yield for interventions at the hospital level include efforts to improve person-centred care, care transition and the experience for those being admitted through the emergency department.
Patient experience has become a focus of the healthcare evolution, and it has been recognised as a key interest to consumers and patient advocacy groups. The Institute of Healthcare Improvement, a leader in the transformation of the healthcare system, has advocated the goal of improving the experience of care within its triple aim of quality.18 The Affordable Care Act in collaboration with the CMS12 has emphasised the need to deliver care that provides a quality patient experience. The act has integrated patient experience scores and reporting mandates into hospital reimbursement strategies which further incentivise excellence. Patient experience scores are reported nationally in the USA,19 and they may be a source of pride and engagement for healthcare teams and used to compete for patients.
The environment is different in Canada as there is currently no financial benefit, and competition between institutions is not a driver for patient services. On the other hand, federal and provincial government health organisations have embraced patient experience as a priority for healthcare, and they have initiated legislation to support its significance in quality delivery. Future public reporting of CPES-IC results and national benchmarking will motivate quality improvement in this area, and patient experience surveying is currently mandatory for hospital accreditation. In Ontario, the Excellent Care for All Act (2010) established that hospitals must develop sustained processes to address and improve the patient experience.20 Our own institution has raised the profile of patient experience to the level of a corporate target by integrating it as a foundation of the vision of the hospital with a priority equal to other quality outcomes and efficiency.
In order to strategise to bring about improvements in patient experience, it is essential to understand how the current American-based survey applies to Canadian culture and our single-payer system. Specifically, it is crucial to appreciate how to adjust for patient demographics within different settings, not just to externally compare with other urban institutions, but also to begin to internally identify factors that may influence overall scoring and interpretation.
The current study is not the first to examine the role of patient and other covariates in the modelling of measures of overall patient experience in Canada.21 However in the latter work, the analysis involved the HCAHPS survey focusing on the single question of rate experience. The authors did demonstrate a similar relationship with higher level of education, urgent admission status and longer length of stay as predictive of poorer measures of experience rating; however, they did not include patient-perceived physical and mental health status, both of which were the most consistent and significant predictors of overall care.
It may not be feasible to generalise from the analysis at a single hospital due to the differing contributions of the patient covariates and interactions with the specific domains of patient care at each hospital across the country.22 For example, race was not found to be a significant factor for most questions unlike in the USA.22 This finding may only be relevant in the context of our centre (a medium-sized Canadian city), whereas it may not apply to larger metropolitan centres such as Toronto and Montreal, where there may be greater ethnic diversity. On the other hand, the finding that women are less likely to provide a topbox scoring on questions of overall experience is in keeping with previous findings with the HCAHPS survey.23
Patient experience key driver analysis has been used to focus attention and initiatives in patient-care areas with high potential to impact on the overall global measures of care. The new CPES-IC survey has been designed to not only include domains currently in the HCAHPS survey, but also domains reflecting patient-centred care, transition of care and the processes of direct or emergency admission. Although these new domains have not been formally validated in the Canadian context, they were all identified as areas of potential high yield in our study in terms of overall contribution to the patient experience. Many of these questions refer to key issues of team communication and the perception of coordination of care; items that could be addressed through team restructuring, checklists and scheduling. On the other hand, nursing and doctor communication skills, though important, did not support targets of high yield in terms of hospital resources.
There are multiple important implications of this work. The analysis highlights the differences in adjusted and unadjusted rankings between departments which emphasise the importance of the use of the demographic covariates obtained from the survey such as perception of physical and mental health and education level. The adjusted improved measures in medicine and family medicine underscore that chronic disease and comorbidity must be taken into account in patient experience initiatives. Recognition of adjusted results also enhances engagement of staff who face the challenges of chronic disease care and provides the opportunity to follow for improvements.
The analysis may be limited by unknown and unmeasured covariates. Only a few of the covariates from the administrative database were significant in models describing perceptions of excellence in individual questions of overall care. Further work will be necessary to determine if these administrative database variables are important at model development at the unit or provider level. Although there was no difference between departments in any of the questions, more subtle comparisons such as between divisions and services may be important in understanding how to advance patient experience initiatives. Finally, patient care domains were not included as covariates in the derivation of the multivariable models for the global overall questions. We elected not to do this as we felt the domains as covariates would demonstrate significant bias due to their correlation not only to the outcomes but also to many of the other predictors. Therefore, we elected rather to look at their interactions and correlations using key driver analysis.
In summary, this analysis provides a perspective on drivers that must be considered when assessing patients’ perceptions on the overall care at a healthcare institution in Canada. Healthcare institutions must incorporate patient demographics and self-reported aspects of perceived health into the analysis of patient experience data to properly interpret this information, particularly when comparing departments and units within the institution. We believe that this understanding will form the basis for a strategy of thoughtful data-driven targeted interventions to improve the patient experience.
References
Footnotes
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval The protocol was reviewed and approved by the Ottawa Health Science Network – Research Ethics Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement There are no additional unpublished data from this study. The study data are partially deanonymised, and therefore, it is kept on a secure server at the Ottawa Hospital.