Article Text
PDF
Abstract
Objectives There remains concern regarding the use of survey data to assess aspects of healthcare quality. The relationship between patient experience and adverse events as documented by patient safety indicators (PSIs) is a timely research topic. The objectives were to document the association of PSIs and patient experience scores, and to determine risk-adjusted odds of high experience scores versus PSI presence.
Setting and participants From April 2011 to March 2014, 25 098 patients completed a telephone survey following discharge from 93 inpatient hospitals in Alberta, Canada.
Research design A modified version of the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) instrument was used. Surveys were linked to inpatient records and PSI presence was documented using a validated algorithm.
Measures Three questions about overall hospital, physician and nurse ratings were scored on an 11-point Likert scale from 0 (worst) to 10 (best). Experience was classified as high (9 or 10) versus low (0–8). Demographic/clinical differences between respondents with/without a PSI were assessed. Logistic regression examined the relationship between factors including PSI and experience ratings.
Results Overall, physician and nurse care was rated high by 61.9%, 73.7% and 66.2% of respondents. 1085 patients (4.3%) had a documented PSI. Most frequent PSIs were haemorrhagic events (n=502; 2.0% of sample), events relating to obstetrics (n=373; 1.5%) and surgical-related events (n=248; 1.0%). Risk-adjusted models showed patients with PSIs had decreased odds of having high overall (OR=0.86; 95% CI 0.75 to 0.97), physician (OR=0.76; 95% CI 0.66 to 0.87) and nurse (OR=0.83; 95% CI 0.73 to 0.94) ratings.
Conclusions There is clear evidence that inpatient experience ratings are associated with PSIs, one element of quality of care. Future research, examining individual PSIs and patient experience questions, is warranted, as this may inform targeted quality improvement initiatives.
- Patient experience
- Patient safety
- HCAHPS
- Acute care
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study examined the association of patient safety indicators (PSIs) and patient experience scores, as documented by the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS), a validated survey.
PSIs were documented using a validated administrative data algorithm. This is a significant advantage over chart reviews, which are time consuming and may be prone to subjective error.
Although administrative data alone may not capture all PSIs, their accepted use as a quality indicator has been documented by several organisations, including the Agency for Healthcare Research and Quality (AHRQ).
The association between patient-reported hospital experience and PSIs lends credibility to the inclusion of patient experience as a reliable, patient-reported account of what occurred in-hospital.
PSIs represent only one aspect of quality of care. Future research which examines the association of patient experience and other aspects of quality of care is warranted.
In recent years, patient-centred care (PCC) has emerged as a key priority for health systems and patients alike. Indeed, the Institute of Medicine considers PCC as one of six key elements of high-quality care.1 Although there is no common definition of PCC, the underlying principle is to engage patients, allowing them to be active participants in their own care. In addition to a clinical emphasis, PCC is the focus of emerging research groups, including the Patient-Centered Outcomes Research Institute (PCORI; USA)2 and the Strategy for Patient-Oriented Research (SPOR; Canada).3
Despite this, there remains scepticism as to whether patients possess the ability to accurately assess the quality of their care. A common method for assessing the perceived quality of healthcare services on the part of patients is to administer a hospital experience survey. In their own right, patient experience surveys offer tremendous value from a quality improvement perspective. Organisations can receive feedback directly from their patients and use the data to guide targeted improvement efforts.4 One drawback of this approach, however, is that surveys are a passive means of assessing quality of care, and that patient experience has been thought to be more reflective of the patient's general mood or subjective response tendencies.5 As gaps in communication may exist between physicians and their patients, it is also acknowledged that patients may not be aware of all medical decisions made on their behalf. In short, when patients report their hospital experience, they may not be making an informed assessment. Thus, evidence to show that patient reports of their hospital experience are associated with other outcomes such as measures of quality of care would help to counter this potential misconception.
Preliminary research has explored the relationship between patient experience and outcomes, with conflicting results. One large, national study showed that a better patient experience was associated with greater inpatient healthcare use, higher overall and prescription drug expenditures, and increased mortality.6 On the other hand, higher patient satisfaction has been associated with better outcomes among those with acute myocardial infarction, congestive heart failure and pneumonia.7–9 It has also been associated with fewer complications10 ,11 and adverse events (AEs).12 Kennedy et al5 found that better patient satisfaction was associated with lower mortality but was not correlated with compliance with process measures or length of stay. A systematic review13 highlighted conflicting results with respect to patient experience and its association with various measures of patient safety. Although it was more common to find positive associations between the two,13 conflicting results may be in part due to variations in the size of the study, the cohort studied (eg, demographics, clinical profile), the context (eg, inpatient, emergency department, primary care) and the methods used to document patient experience.
Although they are similar terms which are oftentimes used interchangeably, it is important to understand the differences between patient satisfaction and patient experience. Jason A. Wolf, President of the Beryl Institute, a global community of practice dedicated to improving the patient experience, states that satisfaction is ‘the idea of how positive someone feels about an encounter’.14 Experience encompasses more than a sense of satisfaction and ‘is defined in all that is perceived, understood and remembered’.14 Patient experience is ‘about ensuring the best in quality, safety and service outcomes’.14 It can assess aspects of PCC such as the inclusion of the patient in care decisions, as well as issues such as patient understanding of their condition/treatment and discharge instructions.
Having standardised methods to document quality of care and patient experience is essential. Patient safety indicators (PSIs) are a validated means to use administrative data in order to document in-hospital AEs.15–21 In the Canadian context, a comprehensive list of PSIs has been developed and validated by our research group, using the Canadian version of the International Classification of Diseases (ICD), 10th revision (ICD-10-CA) (DA Southern, H Quan, WA Ghali. Deriving ICD-10 codes for patient safety indicators for large-scale surveillance using administrative data. Submitted).21 For documenting inpatient hospital experience, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) is a validated, standardised instrument. It is the current gold standard in the USA, where it is mandated under the US Affordable Care Act.22–24 Literature documenting the association between PSIs and patient experience, as documented by HCAHPS, has been non-existent to date. Forster et al25 used a similar survey methodology to demonstrate an association between patient experience and adverse drug events posthospital discharge. This study, however, did not use HCAHPS, but rather an ad hoc survey.
We sought to (1) document the association of PSIs and patient experience scores, as documented by HCAHPS and (2) determine the risk-adjusted odds of high overall, physician and nurse-related experience scores compared with PSI presence.
Methods
Study population
From April 2011 to March 2014, 27 492 respondents completed an HCAHPS-based patient experience survey within 6 weeks of discharge in the province of Alberta, Canada. This number represents 5.6% of the total eligible discharges from the province's 93 acute care inpatient facilities during this period. The survey response rate was 73.3%, as per the following formula:
1
As per the HCAHPS sampling protocol,26 we excluded patients who were under 18 years, had an inpatient stay <24 hours, died during hospital stay, were admitted to a psychiatric unit, had a psychiatric physician consultation or had day surgery or ambulatory procedures. For compassionate reasons, our organisation also excluded visits relating to stillbirths, dilation and curettage (D&C) procedures, or linked to a newborn with length of stay >6 days (eg, complication/neonatal intensive care unit stay).27 A list of eligible patients was generated on a bi-weekly basis from administrative discharge data for each of the 93 hospitals. These data contained up to two telephone contact numbers for each patient, as provided at hospital admission. The data did not differentiate between mobile phones and landlines. Each hospital had a preset monthly quota of complete surveys. This quota corresponded to 5% of eligible discharges.
Survey of inpatient experience
Interviewers followed a standard script with a list of prompts and frequently asked questions and captured data via computer-assisted telephone interview (CATI). Of the 51 survey questions, 32 were from HCAHPS. These items measured nine standard domains: communication with doctors, communication with nurses, responsiveness of hospital staff, pain management, communication about medicines, discharge information, cleanliness of the hospital environment, quietness of the hospital environment and transition of care. Detailed information about the development, validity and American results from HCAHPS is publicly available.23 ,28 The remaining 19 questions addressed organisation-specific policies and procedures such as patient concerns, pharmacy care and patient education. Each survey required 15–20 min to complete.
Interviewers received standard training and conducted random dialling. Each target telephone number was dialled up to nine times on varying days and times. Calls were completed from 9:00 to 21:00 Monday to Friday, and from 10:00 to 16:00 on Saturdays. In total, 10% of phone calls were monitored as per our own institutional and HCAHPS quality assurance standards.26 To ensure responses were based on a specific inpatient visit, each interview began with a verification of the discharge date and hospital name. Respondents were asked to not consider any other healthcare interactions that they may have had during that time. At the end of the survey, patients with a concern, complaint or compliment about their healthcare services were provided with contact information for our organisation's Patient Relations department.
Ethics and consent
Ethical approval for the study was obtained from the Conjoint Health Research Ethics Board (CHREB) at the University of Calgary (file number REB14-2338). A waiver of consent was granted by the ethics board due to retrospective nature of the study. As part of the telephone survey protocol, patients were informed of the possibility that their data could be used for quality assurance and/or research purposes.
Data linkage and defining patient safety indicators
Survey data were linked to the corresponding inpatient discharge abstract data (DAD)29 ,30 using personal health number, facility code and discharge date. A total of 25 098 surveys containing complete data were accurately linked to their corresponding inpatient record—a 91.3% rate. Coders with professional college training on clinical information coding at all hospitals in Alberta coded demographic information, up to 25 diagnoses and 20 procedures from charts after discharge. Diagnoses were coded using the ICD-10-CA system. For each diagnosis, timing of the condition occurrence was also coded. Presence of PSIs was determined using an ICD-10-CA coding algorithm (DA Southern et al. Submitted) containing 17 categories of complications. The algorithm was applied to the DAD to identify diagnoses with ‘type 2’29 and also clinically meaningful patient safety-related events. PSIs were coded as present (one or more events) versus absent (no events). The complete list of specific PSIs which were documented is presented in box 1.
List of documented patient safety indicators (PSIs)
▸ Haemorrhagic events;
▸ Obstetrical complications affecting the mother and/or fetus;
▸ Complications directly related to surgery;
▸ Hospital-acquired infections;
▸ Respiratory complications;
▸ Cardiac complications;
▸ Events proximally threatening to life or to major vital organs;
▸ Gastrointestinal;
▸ Traumatic injuries (non-procedural) arising in hospital;
▸ Central nervous system complications;
▸ Delirium;
▸ Drug-related adverse events;
▸ Adverse events related to fluid management;
▸ Venous thromboembolic events;
▸ Anaesthesia-related complications;
▸ Endocrine and metabolic complications;
▸ Decubitus ulcer.
Study variables
Demographic variables included age group at hospital discharge, sex, marital status, education level and birth location of the patient (Canada vs other). Patient age groups were classified as 18–29 (years), 30–39, 40–49, 50–59, 60–69, 70–79, 80 and older. Marital status was coded as single (never married), married/common law/living with partner and divorced/separated/widowed. Education level was coded as elementary or junior high, senior high, college/technical school, undergraduate level and postgraduate degree complete. Clinical variables were PSI presence, admission type (urgent vs elective), most responsible provider service (family practitioner vs other), discharge disposition (discharged home with/without support vs other) and number of medical comorbidities. Comorbidity profiles were generated according to the Charlson Comorbidity Index31 using a validated administrative data algorithm.32 The number of comorbidities was classified as none, one, two or more.
Dependent variables included three HCAHPS questions pertaining to overall, physician and nurse rating. These three questions were read to patients as follows:
Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible, what number would you use to rate this hospital during your stay?
Using any number from 0 to 10, where 0 is the worst possible doctor care and 10 is the best possible doctor care, what number would you give the care you got from all the doctors who treated you?
Using any number from 0 to 10, where 0 is the worst possible nursing care and 10 is the best possible nursing care, what number would you give the care you got from all the nurses who treated you?
Each question was scored on an 11-point Likert scale from 0 (worst possible) to 10 (best possible). For reporting purposes, responses were classified as high ratings (9–10) (top box) versus low (0–8) (‘middle box’ and ‘bottom box’) ratings. This is concurrent with current HCAHPS reporting standards in the USA, where ‘top box’ represents the most positive response choice(s) for a given question.33
Statistical analysis
Study populations were characterised using descriptive statistics. Frequencies of PSIs were calculated for overall (presence of at least one PSI) and each of the 17 individual PSIs. Demographic and clinical differences between those with and without a PSI were assessed using χ2 analyses. Logistic regression was performed to assess the relationship between PSIs and other demographic/clinical factors, and the overall, physician and nurse top box ratings. All analyses were performed using SAS V.9.3 (SAS Institute, Cary, North Carolina, USA). In all cases, statistical significance was determined a priori as an α level of 0.05.
Results
The mean age of patients was 53.3 years (range=18–101), 65.3% were females, 70.0% were married or living common law/with a partner and 85.7% were born in Canada. The mean length of hospital stay was 5.3 days (median=3.0). A majority of patients were admitted to hospital on an urgent basis (59.8%) and discharged home with or without support (95.4%). Overall, physician and nurse care top box ratings (scores of 9 or 10 out of 10) were given by 61.9%, 73.7% and 66.1% of patients, respectively (figure 1). A total of 1085 patients (4.3%) had at least one documented PSI in their inpatient record. A total of 1914 PSIs were documented. PSIs most frequently documented were haemorrhagic events (n=502; 2.0% of sample), events relating to obstetrics (n=373; 1.5%), surgical-related events (n=248; 1.0%) and infection (n=211; 0.8%). All other PSIs were present in <0.5% of the study cohort. Patients experiencing at least one PSI during their hospital stay were more likely to be female, 18–39 years of age, highly educated and admitted to hospital on an elective basis (table 1).
Demographic and clinical characteristics of sample
Distribution of responses to overall, nurse and physician ratings of care.
Table 2 contains the results of the adjusted logistic regression analyses. For overall experience, having one or more PSIs was associated with decreased odds of reporting an overall top box score. Respondents who were married/common law/living with a partner, those with an education level of college/technical school or less, having a family practitioner as the most responsible provider service and being discharged home with/without support showed increased odds. Decreased odds of having a top box score (ie, having a less than optimal hospital experience) was seen among those 18–69 years of age (compared with 80 years and older), being born in Canada, those admitted on an urgent basis and among those with two or more Charlson comorbidities.
Adjusted ORs (95% CI) for having a high overall, physician and nurse experience (9 or 10 out of 10, ‘top box’ rating) during hospitalisation
For physician experience, having one or more PSIs was associated with decreased odds of a top box score. Conversely, age of 60–69 years, being married/common law/living with partner, an education level of undergraduate level or less, having a family practitioner as the most responsible provider service and being discharged home with/without support had increased odds. Decreased odds of having a top box score was associated with age of 18–59 years, male sex, having been born in Canada, having an urgent admission to hospital and having one or more Charlson comorbidities.
For nursing experience, having one or more PSIs was associated with decreased odds of a top box score. Male sex, being married/common law/living with partner, an education level of senior high or less, a family practitioner as the most responsible provider service and being discharged home with/without support had increased odds. Decreased odds of having a top box score was associated with age of 18–59 years, having an urgent admission to hospital and having one or more Charlson comorbidities.
Figure 2 highlights the odds from stratified analyses, according to gender, age group and number of comorbidities. Most notably, males with a PSI consistently showed decreased odds of having top box scores for overall, physician and nurse care. Females, certain age groups (particularly those 50 years and older) and patients with comorbidities who also had a PSI showed similar decreased odds.
Stratified analyses for PSI presence and ‘top box’ ratings of care, according to gender, age group and number of medical comorbidities. PSI, patient safety indicator.
Discussion
Presence of at least one PSI was associated with decreased odds of having top box HCAHPS ratings of overall, physician and nurse care. This was also shown in risk-adjusted models which controlled for a number of demographic and clinical characteristics. Age, marital status, education level, admission type, most responsible provider service, discharge disposition and number of comorbidities were related to patient experience ratings—replicating previous findings by our group.27 Perhaps most important, our results suggest that when reported as a summarised, system-level performance measure, patient-reported experience is associated with PSIs, one element of quality of care. The association between patient experience and elements of care quality had been shown previously in a study by Isaac et al. They demonstrated that positive experiences were associated with fewer inpatient complications, particularly pressure ulcers, postoperative respiratory failure and pulmonary embolism/deep vein thrombosis.34 Similarly, hospitals with patients who report more positive care experiences have been shown to have employees with more positive perceptions of patient safety culture.35–37 Our study expands upon these findings, using a validated algorithm for documenting a wide range of PSIs.32 Additionally, our results had not been previously demonstrated using an HCAHPS-based instrument in a Canadian setting—one with universal Medicare coverage.
We suggest that a standardised measure of patient experience should be used as an indicator of PCC and to monitor healthcare system performance. This is an area of research that has been to date largely untouched. One advantage of patient experience, as captured via HCAHPS, is that a direct report is provided by the patient using a validated instrument. This provides opportunities for valid comparisons across hospitals and healthcare organisations, particularly when using case-mix and mode adjustment to account for demographic, survey administration (eg, mail vs phone) and service-level differences.38 It should also be noted that the HCAHPS validation process used patients from the outset—allowing for an accurate reflection of what is deemed important from patients themselves.
There are many opportunities for future use of inpatient experience data. Communication between clinicians and patients plays an important role in PCC. This reflects a somewhat fundamental change in the perspective of physician–patient interaction. Within the context of PCC, physicians do not make treatment decisions on behalf of the patient, but rather in conjunction with the patient. This encourages transparency as well as the incorporation of the patient's values, beliefs and choices throughout their care journey. In their review of patient perceptions of healthcare quality, Sofaer and Firminger conclude with the following statement: ‘If we are truly to achieve a healthcare system that is patient-centered, we must continue to search for creative ways to elicit, and heed, the voice of the patient’.39
The present study has several strengths. It is the first to link Canadian inpatient experience data to PSIs using an ICD-10-CA algorithm. In their 2013 commentary, Manary et al40 made a series of recommendations to further validate comparisons of patient experience and outcomes. These were that future said comparisons should (1) focus on a specific event or visit, (2) focus on patient–provider interactions, (3) ensure the timeliness of the measure to limit recall bias and (4) perform risk adjustment. The present project satisfies all four of these criteria.
Another strength is that the survey was conducted using a validated instrument (eg, HCAHPS), with a standard script, prompts and answers to frequently asked questions. These help ensure the highest degree of standardisation and reliability, as compared to historical investigations of patient experience, which have primarily used ad hoc instruments.
Additionally, the quality and breadth of our abstracted data is a tremendous asset. As the sole provider of provincial inpatient healthcare services, Alberta Health Services has complete documentation on all inpatient visits that occur in our jurisdiction. Thus, the potential for data linkage is great as no gaps in data coverage will occur. This overcomes a huge limitation present in other jurisdictions that do not have a universal healthcare model.
The final study strength lies within our comprehensive survey sampling strategy. As opposed to cherry-picking patients, the sample is derived from all eligible inpatient discharges. Thus, each potential participant has an equal chance of participation, regardless of institution, date of service or clinical condition. Contact information includes up to two telephone numbers provided at the time of hospital registration, thus are presumed to be the most accurate way of contacting patients. Contact is attempted up to nine times at varying times over varying days, including one weekend day. Patients unable to speak freely are provided with an opportunity to book a call-back time, at their convenience. Our high response rate (73%) and representativeness of the sample41 demonstrate the success of these strategies.
There are some limitations to the present study which warrant discussion. The first is that PSI represents only one aspect of quality of care. Other aspects (eg, medication adherence, readmission rate) may have a different relationship with patient experience. Second, although administrative data alone may not capture all PSIs21 (DA Southern, et al. Submitted), several validation studies document their accepted use as a quality indicator, including ones by the Agency for Healthcare Research and Quality (AHRQ). Third, it has been postulated that to accurately obtain an educated assessment of patient experience, it is necessary to educate patients a priori regarding appropriate expectations of care.42 In our opinion, we feel that this would be an excellent topic for future research. Fourth, due to the cross-sectional nature of our study, we advocate caution in interpreting the study results. These should be considered as associative only, and causality should not be inferred. As in previous work by our group,27 there were many other factors (eg, demographic, clinical) that were associated with high experience ratings. Although these were controlled for in the present study, we did not perform any case-mix adjustment, as is done in the USA.43 Last, as this was a Canadian study, results may vary in other jurisdictions, particularly those with differing healthcare models (eg, UK, USA).
In conclusion, the present study demonstrates a clear association between patient-reported hospital experience and an element of healthcare quality, via documentation of PSIs using administrative data. The study has a clear policy implication, as we have demonstrated that subjective patient accounts are associated with an objective element of care quality. Showing that patients can accurately report what took place in hospital lends further support to the inclusion of patient experience as a measure of health system performance. This also supports the documentation of patient experience for quality improvement purposes. Future research, examining individual PSIs and specific patient experience questions, is warranted, as certain aspects of care may be closely associated with AEs. The association of other aspects of quality of care with patient experience should also be examined. Last, future studies which include in-depth interviews and a measure of patient expectations may provide additional insight regarding how patients rate their hospital experience.
References
Footnotes
Acknowledgements The authors acknowledge the team of Health Research Interviewers from Primary Data Support, Analytics (DIMR), Alberta Health Services, as well as the patients who completed the survey.
Contributors KAK, MJS and HQ developed the research question and study methods. Data collection, linkage and analysis were performed by KAK, BMC, MJS and HQ. All authors contributed to the drafting and editing of the manuscript, including approval of the final version submitted for publication.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Ethics approval University of Calgary Conjoint Health Research Ethics Board (CHREB) approved the study and provided a waiver of consent (file number REB14-2338).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.