Article Text
PDF
Abstract
Objective In Alberta, the Alberta Paediatric Inpatient Experience Survey (APIES) is used as a proxy-reported measure of paediatric experience. To our knowledge, the influence of casemix factors on patient experience as measured by paediatric patient experience surveys have not been reported within Canadian paediatric samples. In this paper, we sought to determine the patient and respondent factors associated with paediatric inpatient experiences in Alberta, Canada.
Design Retrospective analysis of patient experience survey data.
Setting Inpatiet acute care hospitals in Alberta, Canada.
Intervention and main outcome measures Retrospective analyses were conducted using APIES surveys linked with eligible inpatient records (n=6262). Descriptive statistics were reported. χ2 tests were performed to assess distribution of casemix between general and paediatric hospitals. Logistic regression was performed with overall hospital experience as the dependent variable with casemix and hospital variables as independent variables.
Results Casemix characteristics were unevenly distributed between general and paediatric hospitals. Compared with reference categories, older respondents, healthier patients and treatment at paediatric facilities had increased odds of providing most-positive ratings. Increased respondent education was associated with decreased odds of providing most-positive ratings. Likelihood-ratio tests showed that most casemix variables improved model fit, except for respondent relationship to the patient.
Conclusions To improve reports of paediatric inpatient experience, administrators and providers require reliable and comparable measurement. Both the Child Hospital Consumer Assessment of Healthcare Providers and Systems and other measures of patient and family experience need to consider patient and respondent characteristics when interpreting results. Considered with other research from patient experience in Alberta, we discuss future directions and quality improvement implications.
- PAEDIATRICS
- Quality in health care
- Health policy
- STATISTICS & RESEARCH METHODS
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study demonstrates the need for casemix adjustment for paediatric experience surveys in a Canadian setting.
We call for increased measurement of paediatric patient experience within Canada and discuss avenues for additional research and collaboration.
Some demographic factors were not available for this data, and surveys of paediatric inpatient experience rely on proxy reports from parents or caregivers.
Introduction
Patient-reported experience measures (PREMs) are central to measuring the provision of patient-centred care, a conception of healthcare systems that prioritises the experience and input of patients in managing their health.1–3 PREMs may serve as an important data source for health quality improvement and for understanding how specific aspects of care relate to overall experiences. Therefore, PREMs may be able to provide actionable direction for efforts to improve healthcare systems.4 5 The Child Hospital Consumer Assessment of Healthcare Providers and Systems (Child-HCAHPS) is a PREM developed to provide standardised, comparable assessments of proxy-reported paediatric inpatient experiences.6 It has been adopted across several jurisdictions7–10 to assess these experiences. In Canada, paediatric patient experience is not uniformly assessed using the Child-HCAHPS. Paediatric patient experience data has been collected by healthcare systems in at least three Canadian provinces (British Columbia, Alberta, Ontario) but is not systematically reported or compared across them.11–13 To our knowledge, Alberta is unique in that Alberta Health Services (AHS, the provincial health authority) regularly reports on adult and child patient experience.13
Patient experience is not uniform, as patients may report overall positive or negative experiences based on different aspects of their health or care. However, they may also vary by patient-level factors. Research from the USA has shown that such factors include patient age, health status, respondent age, respondent relationship to the patient and respondent level of education.10 However, there has been limited research validating casemix adjustment for the Child-HCAHPS within Canada.8 9 In Canada, the Canadian Patient Experiences Reporting System is a national system set up by the Canadian Institute for Health Information (CIHI) allows for between-jurisdiction comparisons on the Canadian Patient Experience Survey-Inpatient Component (a CAHPS-derived survey of adult experiences). However, there is not yet a comparable system set up for paediatric experience surveys. To understand the importance of casemix adjustment in paediatric inpatient experience within Canada, analyses of these factors on hospital experience could build on assessments from research on patient experience surveys in the USA.10 Bridging this gap is an important step in demonstrating the comparability of Child-HCAHPS-derived surveys in the Canadian setting, to allow for targeted quality improvement interventions that lead to better overall experiences, and to enable further research on patient and family care.
To understand the drivers of positive paediatric hospital experiences across Alberta, the current study looks at the relationship between individual patient and respondent characteristics with overall ratings of hospital experience as measured on the Alberta Paediatric Inpatient Experience Survey (APIES) (adapted from the Child Hospital CAHPS Survey). The objectives of this retrospective analysis of patient experience data are to assess the casemix determinants of a positive paediatric inpatient experience and to describe the distribution of casemix between paediatric and general hospitals.
Methods
Study population
Telephone surveys were collected between 2015 and 2019 by AHS, the provincial health authority and soleprovider of inpatient care in Alberta, serving approximately 4.4 million residents. Respondents were parents or guardians of paediatric patients who spent at least 24 hours in an Alberta hospital.
An existing data sharing agreement in place between the research team at the University of Calgary and AHS allowed us to use these data.
Survey administration
Detailed eligibility criteria for the survey have been previously published.8 9 Consistent with published guidelines,6 10 AHS excludes some patients from sampling. Respondents were eligible to participate if their child was hospitalised on an inpatient basis for a minimum of 24 hours, was under the age of 18 at hospital discharge, did not spend stays entirely within the emergency department, did not present for mental health concerns and was alive at discharge. Surveys in this study’s sample were administered from discharges at 13 hospitals across the province (1 stand-alone paediatric hospital, 1 paediatric hospital within a larger hospital and 11 primarily adult sites which also see a large volume of paediatric patients). AHS also collects data from respondents reporting on the experiences of newborns, but these cases were excluded from the analyses. Previously published guidelines note the confounding factor of the mother’s care in ratings of newborn experiences.6 Cases were also excluded if less than 100 patients were sampled from the hospitals.14
Respondents were contacted by telephone call, up to 6 weeks postdischarge. To ensure the greatest opportunity to participate, phone calls were conducted on varying days and times (weekdays 9:00–21:00 hours, 10:00–15:00 hours on Saturdays). Each eligible number was dialled up to nine times. If desired, once contacted, respondents could request a call-back time to complete the survey at their convenience. AHS reports a high response rate (67% for 2019/2020 fiscal year; note: we do not have access to response rates for each year).
Alberta paediatric inpatient experience survey
Toomey et al6 developed a paediatric version of the HHCAHPS, a tool developed in the USA to evaluate American hospital performance from the perspective of patients. The Child-HCAHPS was developed as a ‘standardised survey of inpatient experience’ for paediatric populations.6 As previously reported, AHS continuously collects data using an augmented version of the Child-HCAHPS known as the APIES.8 This includes 39 experience and 10 screening questions from the original Child-HCAHPS, as well as additional patient experience and screening questions developed for organisational purposes. Respondents are asked to rate their child’s overall health, previous hospital stays and provide additional demographic information. For the purposes of our analyses, we have only included measures from the original Child-HCAHPS to compare our results directly with similar studies.15 The modified survey used by AHS is included in online supplemental appendix A.
Supplemental material
The Child-HCAHPS (and APIES) asks respondents about aspects of inpatient experience including communication with parent/guardian, communication with child, attention to safety and comfort, hospital environment and global ratings of hospital quality.6 Questions have variable response scales. Some (eg, ‘During this hospital stay, how often did your child’s nurses treat you with courtesy and respect?’) are answered with a four-point scale: ‘never; sometimes; usually; always’. Others (eg, ‘During this hospital stay, did providers or other hospital staff tell you how to report if you had any concerns about mistakes in your child’s care?’) are answered with ‘yes, definitely’; ‘yes, somewhat’ or ‘no’. Questions assessing overall experiences (overall care, care from nurses and care from doctors) are rated on an eleven-point numerical scale, ranging from 0 (worst possible) to 10 (best possible).
Clinical records
Data were linked via provincial personal health number to the corresponding inpatient records, obtained from the Discharge Abstract Database (DAD).16 The DAD provides individual-level clinical and demographic information about patients discharged from acute care (typically hospitals). Child age at discharge was the only DAD variable included in the regression model.
Patient and public involvement
Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.
Statistical analysis
Data were initially cleaned and linked by AHS. The research team performed additional cleaning to compute top-box scores (a binary variable delineating the highest possible ratings vs all other responses) and assign casemix categories. Responses where respondents reported not knowing the answer, refused to answer or were otherwise missing were recoded as ‘not available’ (NA) prior to analysis. All analyses and cleaning were performed with R V.3.6.3. A uniform significance level of 0.05 was assigned for all tests.
Descriptive statistics of the sample were computed. χ2 tests were run to assess the distribution of covariate factors (discussed below) by hospital type (general or paediatric).
Responses to the question assessing overall hospital experience (‘Using any number from 0 to 10, where 0 is the worst hospital possible and 10 is the best hospital possible, what number would you use to rate this hospital during your child’s stay?’) were dichotomised as ‘top-box’ (the highest possible ratings of experience, 9 and 10) vs other ratings (representing varying rooms for improvement, 0–8). Grouping the highest rated experiences into this ‘top-box’ is a common practice in analyses of HCAHPS data.17
Casemix adjustment is typically performed on CAHPS surveys to account for sampling differences between hospitals.18–20 Patient (age, respondent-reported health status) and respondent (age, education, relationship to child, preferred language) characteristics are included as casemix adjustment variables for the Child-HCAHPS.6 Results should also be adjusted for hospital characteristics (eg, approximate volume of cases seen, service types), when available.16 These details were unavailable and could not be counted for in modelling. In our models, we chose to use the casemix factors identified in earlier Child-HCAHPS research from the USA, and added hospital type to specifically account for the differences in reported experiences between hospital type as described by Toomey et al.10 Corrections for survey-mode21 were not necessary, as data were only collected by telephone.
Logistic regression
A multivariable logistic regression model was constructed to assess potential casemix variables (identified by Toomey et al6) and hospital type (identified by Toomey et al10) as predictors for overall hospital experience rating. The binary dependent variable was the top-box overall rating of the hospital experience, with remaining scores (0–8) set as the reference category. The casemix predictor variables included child age (<1, 1–4, 5–8, 9–12,≥13 years), respondent ratings of child’s health (poor, fair, good, very good, excellent), respondent age (<25, 25–34, 35–44, ≥45 years), respondent relationship to child (mother, father, other) and respondent education (≤8th grade, some high school, high school or equivalent, college or non-university certificate/diploma, 4 years university degree, >4 years university degree; refer to online supplemental appendix A for more information).8 Though the survey responses offer choices beyond these categories, they were chosen to be consistent with categories used in casemix adjustment determined by the developers of the Child-HCAHPS.10 Hospital type (general, paediatric) was entered as a fixed factor in the model. ORs and corresponding 95% CIs were reported for each factor. Likelihood-ratio (LR) tests were conducted for each of the factors in the model to assess whether each potential casemix variable improved the models.22 For these LR tests, the potential casemix variables were entered in a model and compared against a model without each potential casemix variable, in turn (eg, the full six-factor model was compared with a five-factor model to determine whether including the variable could improve the overall regression model). The analysis plan was informed by previous Canadian research by Rubens et al.22 Only complete cases were included in the regression models and LR tests; imputation was not performed.
Results
Between October 2015 and March 2019, 7951 surveys were administered. After excluding newborn (n=1362, 17.1%) and at facilities with <100 surveys (n=327, 4.1%), the resulting sample included 6262 cases.
Sample characteristics and χ2 results for the distribution of covariates by hospital type are reported in table 1. Our sample was disproportionately from the province’s two children’s hospitals (n=4401; 70.3%). Female patients comprised 45.5% of cases (n=2848). Gender characteristics are NA for this sample. Unplanned (urgent) admissions comprised 75.1% of the sample (n=4704). Respondents were primarily mothers (n=5274; 84.2%), followed by fathers (n=748; 12.0%) and all other caregivers (n=240; 3.8%). Most respondents (n=5847; 93.4%) had at least a high school education, and almost half of those had university or advanced degrees (n=2586; 40.3%). This is reflective of the latest Alberta census results, which estimated that 90% of Albertans between the ages of 18 and 64 had completed high school.23 Many survey respondents rated their hospital experience a 10 out of 10 (n=2570, 41.0%), and most (66.1%) of responses were either a 9 or 10. The results of the χ2 tests showed statistically significant different distributions for all casemix covariates by hospital type (p<0.01).
Distribution of casemix adjustment covariates by hospital type
Table 2 displays the results of the multivariable logistic regression model for casemix factors and hospital type.22 A total of 6069 complete cases were included in the regression models. Assessment of variance inflation factor showed no multicollinearity. All lLR tests were statistically significant except for respondent relationship to the patient. Given the small reference category for respondent level of education (n=58, or 0.9% of the sample) a second multivariable logistic regression model was conducted where the two lowest level of education were collapsed into a single level as the reference category. The significance of each variables level changes, but ultimately the LR tests remain significant for the same variables. The results of this multivariable regression are available as online supplemental appendix B.
Supplemental material
Impact of casemix adjustment variables and hospital type on overall hospital experience rating
Discussion
This paper expands on previous descriptive work looking at inpatient experiences in Alberta, Canada8 9 by examining the association of casemix factors on overall hospital rating in a large paediatric dataset. We sought to assess whether casemix variables common to Child-HCAHPS analyses in the USA impacted the overall experiences in a Canadian province and examine the distribution of casemix variables between general and paediatric hospitals.6 10 15 22 Understanding the factors influencing top-box score ratings provides direction to healthcare quality improvement initiatives and can allow for improved patient experience. Importantly, adjustments for casemix allow for between-hospital comparisons when modelled hierarchically. Through our pooled analyses and LR tests, we found that casemix variables identified in the United States have applicability for the province of Alberta.
Overall, most (66.1%) respondents rated the overall hospital experience with a top-box score. We found that the ages of both the respondent and the patient, the respondent’s level of education, and the respondent assessment of patient health were all found to be associated with respondents' overall ratings of hospital quality. Consistent with previous research,10 we found that respondents reported more positive overall experience when they were seen at paediatric hospitals when compared with those seen at sites that primarily treat adults.
These results also provide direction for focused assessment of respondent (family or caregiver) experiences with care. Consistent with patient experience research, casemix variables identified had an impact on the odds of reporting most-positive experiences. Older respondents and respondents for healthier patients were more likely to provide most-positive ratings, and respondents with more years of education were less likely to report most-positive experiences.22 These results suggest a need for further investigation of differences between respondent subgroups.
Considered together, these results suggest that children with lower levels of health overall may require a different care experience design than those with better health status. The fact that treatment within paediatric facilities was associated with higher overall scores suggests that more general facilities can improve their Child-HCAHPS scores (and paediatric patient experience) by ensuring that their care of these patients is informed by paediatric hospitals where possible, potentially with a provision of resources or the development of a culture specific to paediatric care. Comparison of practices between general and paediatric facilities may be warranted within Alberta.
The results of the χ2 tests found unequal distribution of every casemix covariate between general and paediatric settings. While the adjusted and unadjusted models testing the drivers of positive overall experiences have been largely similar in previous reports,8 9 results from other jurisdictions suggest that casemix adjustment can have small but meaningful effects on the interpretation of CAHPS-derived surveys.19 22 LR tests allowed for assessment of improved model fit for each of the casemix variables. We found that each casemix variable was associated with a statistically significant fit improvement, except for the respondent’s relationship to the patient. This exception was surprising, as parent identity has been identified as an important factor given the different roles and experiences of parents during inpatient stays.6 Despite the lack of significance as a potential casemix factor, research using Canadian data that may have implications in non-Canadian settings should still consider adjusting for respondent relationship to child for comparability between jurisdictions.
When considered with those in other analyses of paediatric experience analyses in Alberta,8 9 this paper provides evidence to suggest that research with the Child-HCAHPS in Canadian settings may be able to provide comparable data for quality improvement research and practices when utilising casemix adjustment developed in the USA. While comparisons between the Canadian and American healthcare quality improvement contexts should be drawn carefully, we believe the research and insights produced with the Child-HCAHPS in either country can inform quality improvement projects in both. In the USA, public reporting of the HCAHPS surveys has led to improvements in patient experience.24–26 The Triple Aim framework of healthcare quality improvement seeks to improve patient experience, improve population health and reduce the cost of care.27 Increased data collection and public reporting of inpatient paediatric experiences may provide improved care in the Canadian setting and assist in meeting Triple Aim goals.
To our knowledge, Alberta and Ontario are the only two Canadian provinces routinely collecting paediatric patient experience with modified Child-HCAHPS instruments.11 12 The CIHI has a reporting system and comparison tool for the Canadian Patient Experience Survey (an HCAHPS-derived instrument)28 that allows hospitals to compare and analyse patient-reported hospital experience among adults. CIHI will begin publicly reporting these results in 2022.29 It is our hope that measurement of paediatric patient experience will become as universal as adult experiences to improve paediatric inpatient experience in Canada.
There are several strengths and limitations to this study. Assessing patient experiences in a paediatric population can be challenging for several reasons, including the issue of using proxies (primarily parents) to report the patient experience30 and ethical concerns with directly surveying children.31 While our sample size was large, respondents disproportionately tended to be mothers (consistent with previous Child-HCAHPS research).10 Future work is needed to explore how respondent characteristics and respondent-patient relationships may influence responses on proxy PREMs. Additionally, the APIES is conducted via telephone between 2 days and 42 days postdischarge. While operators ask respondents to confirm the child’s specific hospital stay as part of the APIES administration, previous hospitalisations, changes in patient health and other events may have biased respondents’ experiences. We could not account for some environmental factors within hospitals (such as accommodation for parents) which are known to play a role in patient experience and outcomes.32 33 Another limitation of this paper was the lack of additional features about the hospital and clinical stay (such as a facility’s volume of cases, service type). Our limited data sharing agreement did not provide details for all elements from the DAD; additional data linkage is warranted to investigate the impact of these factors in the paediatric setting. Additionally, our dataset lacked certain socioeconomic variables, such as ethnicity, income and other factors known to be associated with perceptions of care and responses to patient experience surveys.34–37 Some of these were unavailable within the DAD, and ethnicity was not provided by the provincial health authority. Furthermore, surveys were only conducted in English. There is considerable variability within patient populations that has not been adequately addressed by this paper, but it is our hope that future subgroup analyses will allow for more targeted quality improvement.24
Future analysis should seek to understand patient experience while considering children’s medical complexity, respondent characteristics and intrahospital variability in the drivers of paediatric inpatient experience. Larger samples, multilevel modelling and routine data collection across provinces would allow for an assessment of the impact of casemix variables across Canadian jurisdictions and inpatient settings. We are exploring opportunities for additional linkages of hospital and clinical characteristics to study these issues.
Conclusion
Person-centred healthcare is increasingly being recognised as an important determinant of health outcomes and of patient experience. Reliable and comparable measurements are needed when studying patient experiences and person-centred healthcare. Within Canada, measures derived from the Child-HCAHPS scores should be adjusted to provide for equitable comparisons between hospitals. Beyond CAHPS-derived surveys, this study underlines the importance of assessing both respondent and patient characteristics within different care settings to understand patient and family experiences of care.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
This project received research ethics approval from the University of Calgary (REB17-0769). Due to the retrospective nature of our project, a waiver of consent was granted.
Acknowledgments
We would like to thank Primary Data Support, Analytics Department, Alberta Health Services for collection of data.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @brn_stl, @MariaJ_Santana
Contributors KK and M-JS conceived the research direction with input from BJS and PF. BJS, KK, and PF conducted the analyses. BJS and PF drafted the manuscript, and all named authors contributed substantially to the development of the manuscript and its revision. All authors agreed to be accountable for all aspects of this work; M-JS is the guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.