Article Text
PDF
Abstract
Background Patients seeking emergency and urgent care tend to experience a system, making choices about which service to use and making use of a number of services within a healthcare episode. The aim was to psychometrically test the Urgent Care System Questionnaire (UCSQ) for the routine measurement of the patient perspective of the emergency and urgent care system.
Methods The UCSQ was developed based on qualitative research with recent users of the system. It consisted of a screening question to identify recent users and questions on the patient experience of, and satisfaction with, their most recent event. The acceptability, validity and reliability of the UCSQ were tested in a postal survey of 900 members of the general population and a telephone survey of a quota sample of 1000 members of the general population.
Results The response rate to the postal survey was 51% (457/893). In the telephone survey, 11 604 calls were made to obtain a quota sample of 1014 people. These surveys identified 250 system users in the previous 3 months. A principal-components analysis identified three satisfaction components with good internal consistency (Cronbach alpha between 0.7 and 0.93): ‘progress through the system’ (10 items), ‘entry into the system’ (three items) and ‘patient convenience’ (five items). These components varied as expected by age and overall rating of the system.
Conclusion Preliminary testing suggests that the UCSQ has reasonable acceptability, validity and reliability. Further testing is required, particularly its responsiveness to changes in emergency and urgent care systems.
- Patient satisfaction
- healthcare quality
- access and evaluation
- emergencies
Statistics from Altmetric.com
A recent review of the National Health Service (NHS) in England places the patient experience at the heart of healthcare provision.1 This creates an increasing imperative for the routine measurement of the patient experience of healthcare. Emergency and urgent care, where patients perceive the need for immediate access to advice or treatment,2 is a large and complex part of healthcare delivery. There are many access points to emergency or urgent care in England including emergency ambulance, emergency departments, daytime general practice, out-of-hours general practice, walk-in centres, minor injury units, NHS Direct the 24 h telephone helpline, and pharmacy. It is usual for patients to use a number of these services within a healthcare episode.3 Thus, patients tend to experience a system rather than a single service, making choices about where to enter the system, and then moving through a series of services which may feel coordinated or leave users feeling bounced around, or stuck within, this system.4
There are many instruments which assess the patient experience of, or satisfaction with, different services within the emergency and urgent care system such as emergency ambulances,5 out-of-hours primary care,6–8 and in-hours general practice including same day appointments for urgent care.9 There is no instrument which attempts to measure the patient experience and view of the emergency and urgent care system, and so we undertook qualitative research with people who had recently sought healthcare urgently to understand the system from their perspective.4 We found that dimensions covered by service-specific questionnaires are of relevance to the emergency and urgent care system, and that the system also has its own characteristics. Issues important to patients included informational continuity, knowing the most appropriate place to enter the system and coordination between services during an episode of care. Some of these have been highlighted in a study of the patient perspective of the primary–secondary care interface, which is a significant part of the wider healthcare system.10
Measurement of healthcare performance can be undertaken from different perspectives—for example, by academics as part of their evaluative research, and patient groups, healthcare commissioners and service providers as part of the routine monitoring of the quality of healthcare. Our goal was to develop a survey methodology which would be feasible in the context of commissioners and system providers routinely measuring the performance of their system from the patient perspective. We faced three challenges. The first was that there is no record of system users in the way that there would be for users of a single service. One way of identifying system users would be to identify users of all the individual services in the system over a specified time period. However, these lists of users from individual services would have many duplicates because so many people use two or three services in an episode of urgent healthcare. An alternative solution is to screen the general population to identify recent users of the system and then ask for details of the most recent contact from the users. We opted for the latter approach because it did not suffer from the problem of duplicates and could identify people who had attempted to contact a service and failed. The second challenge was how best to administer the survey, and we tested a postal approach versus a telephone approach. The third challenge was to develop and test a questionnaire to measure user experience and views of the emergency and urgent care system. This third challenge is reported in this paper. We developed a questionnaire for use with the general population which would include a screening question to identify recent system users and then detailed questions about recent use. The questionnaire was based on our previous qualitative research,4 and within this paper we report testing of its acceptability, validity and reliability in the context of routine performance measurement in the health service.
Methods
Questionnaire development
We developed a questionnaire for completion by members of the general population. A screening question identified users of the emergency and urgent care system in the previous 3 months. For recent users, the questionnaire addressed patient-reported system metrics, descriptive aspects of patients' experiences and patients' views of those experiences. The metrics were constructed by our research team; two key metrics were the number of services used in an episode of urgent care and the length of time from the beginning of an episode until definitive care. Descriptions of experience were also determined by our research team and included details of the first three services contacted and reasons for moving from one service to another. Questions about patients' views or satisfaction with the system were derived from three sources. First, we used items from service-specific questionnaires which we felt were relevant to patients' views of the system. Second, we used items which we felt should be relevant to system experience—for example, sharing of information between services. We produced a draft questionnaire and cognitively tested it within the interviews and focus groups undertaken to explore patients' views of the system.4 The questionnaire was refined throughout this process. Third, an influential source of questions was the analysis of our qualitative data, from which we identified 22 items which captured patients' views of the system.4 No consistent response set has been used in patient satisfaction surveys for services in the system. Therefore, we selected a five-point Likert scale, ‘strongly agree, agree, unsure, disagree, strongly disagree,’ for the 22 items because it allowed us to word items in a way which captured the voices of participants in our earlier qualitative research. The postal questionnaire consisted of 57 questions on 12 sides of A4 (see electronic version). The telephone questionnaire used the same questions with a script developed for the telephone administration.
Data collection
We used two modes of administration for the general population survey—postal and telephone—as part of a wider study of testing the most feasible survey methodology for healthcare commissioners to use routinely when measuring the patient perspective of the system. We undertook both surveys within a single emergency and urgent care system which is managed by an urgent care board of commissioners and service providers. The system covered a population of one million people in a geographical area spanning two primary care trusts in England.
For the postal survey, we used general practice lists to identify 1000 members of the general population. We planned to select a geographically stratified random sample of 20 practices from the 134 in the system, with 50 patients randomly selected from each practice. Recruitment of practices proved difficult, and we increased our request from a sample of 50 patients to a sample of 100 patients from practices recruited later in the process. In practice, 13 participating practices selected a random sample of between 50 and 100 patients from their lists. Questionnaires were sent to adults aged 16 and over, and the parent/guardian of children aged up to 16 years of age. Two reminders were sent to non-respondents.
For the telephone survey, we engaged a market research company to undertake a standard market research survey using random digit dialling and quota sampling to obtain 1000 responses representative of the age and sex of the system population, as used in other health surveys.11 12 To do this, the market research company identified the postal sectors within the area and undertook random digit dialling within these to identify people willing to take part in the survey. Once someone agreed to take part, they were asked for their age and gender. If someone was still needed to fulfil the age gender quota, the person continued with the survey. If the age gender group they fell into was filled, then they were thanked for their help, and the survey was not continued.
Analysis
We tested the acceptability, validity and reliability of the questionnaire.13 The acceptability was tested by considering survey response rates, item response rates and response distributions for the 22 satisfaction items. We expected the rate of missing responses to be higher for the postal survey than the telephone survey,14 and we expected skewed response distributions for satisfaction items because satisfaction with healthcare is always positively skewed. Content validity was derived from basing the questionnaire on qualitative research. Face validity was assessed partly by cognitive testing in the earlier qualitative research and partly by checking for consistency of answers within each questionnaire. Exploratory factor analysis was undertaken on the 22 satisfaction items to test construct validity. We used SPSS principal-component analysis with varimax rotation and eigenvalues greater than 1 to identify domains of system satisfaction.15 Reliability was tested by measuring the internal consistency of domains using Cronbach alpha, requiring values to lie between 0.7 and 0.9. Precision was addressed using a five-point Likert scale for the 22 satisfaction items. Domain scores were calculated by scoring individual items from ‘strongly agree=5’ through to ‘strongly disagree=1’ for positive statements, with reversal for negative statements. The mean score in each domain was calculated so that scores varied between 1 and 5, where 5 indicated the most satisfaction. Missing values for individual items within a domain were replaced by the mean for that domain. Finally, we further tested the construct validity of the instrument by examining domain scores across subgroups known to hold different views of services. Specifically, older people are more likely to be satisfied with services than younger people.16 The relationship between domain scores and overall satisfaction with care was also tested using ANOVA. All analyses were undertaken in SPSS version 12.1.
Results
Acceptability
The response rate to the postal survey was 51% (457/893) when 7 ‘return to senders’ were removed from the denominator. The aim of the telephone survey was to identify a quota sample of 1000 respondents representative of the age and gender of the population of the area covered by the system. A total of 11 604 calls were made to identify 1014 respondents. Ninety-nine and 151 respondents reported being users of the system in the previous 3 months in the postal and telephone surveys respectively (total=250). The percentage of missing values for items on the telephone survey varied between 0 and 4%. Most items on the postal questionnaire had missing values in the same range of 0–4%, but percentages were much higher for the satisfaction items, with four items having large percentages between 12% and 18%. A study of individual questionnaires revealed that some respondents put ‘N/A’ against the items with high numbers of missing values, highlighting the need for a ‘does not apply’ option in the response set. The distributions of responses for satisfaction items were as expected for all 22 satisfaction items, with skewed distributions, but use of the full response set (table 1). Response patterns were generally very similar for both surveys, with one exception. There was a tendency for the positive skew to be stronger for the telephone survey than the postal survey, possibly due to social desirability bias. Some items had high use of the middle category of the response set for example ‘services understood that I had responsibilities, like my need to look after my family’ (table 1). This further highlighted the need to add a ‘does not apply’ option to the response set because these items were relevant to some people only.
Distribution of responses to satisfaction items in the Urgent Care System Questionnaire
Validity
We read individual completed postal questionnaires to look for inconsistencies and comments which respondents had made beside questions. A key inconsistency occurred around the types of service contacted. Respondents sometimes ticked ‘Accident and Emergency’ in one question but ‘Minor Injury Unit’ in another where they were expected to tick the same service. This highlighted that people do not necessarily understand which service they have accessed because of the increasing complexity of services within the system and the names by which they are known.4
We undertook an exploratory factor analysis on the 22 satisfaction items using the 250 responses from both surveys. This gave an adequately sized dataset of over 10 subjects per item tested. We removed one item ‘I got the help I wanted quickly,’ which was too highly correlated with two other items (r≥0.8 item correlation matrix).15 Analysis of the 21 remaining items identified three components with eigenvalues ≥1, accounting for 63% of the variance (table 2). The first component contained items relating to the progress people made through the system and consisted of 13 items explaining 37% of the variance. The second component related to entry into the system with three items explaining 14% of the variance. The third component, related to patient convenience, consisted of five items explaining 12% of the variance. Factor loadings, that is, the correlation of each item with the principal component,15 were greater than 0.4 for all items. Some items loaded onto two factors. We undertook an oblique rotation Oblimin which identified that the two items of ‘progress’ which loaded onto two factors with Varimax clearly loaded onto factor 1 only. However, the other items which loaded onto two factors with Varimax continued to do so with Oblimin. Therefore, we allocated them to the component where they best fit the underlying construct.
Results of factor analysis using principal-components analysis on 21 items (N=250; all factor loadings ≥0.4 shown)
For each component, the scores were distributed across the full possible range but skewed towards higher scores, particularly ‘entry’ (table 3).
Distribution of component scores
Reliability
Cronbach alpha was in the acceptable region of 0.7–0.9 for two components, but higher than recommended for the 13 item factor (table 2).15 The Cronbach alpha is positively associated with the number of items within a component. Removal of any single item did not reduce the Cronbach alpha for this component. We used the interitem correlation matrix to find correlated items and removed three of these to reduce the Cronbach alpha to 0.93 for the remaining ten items, which was still higher than recommended.
Further construct validity
Component scores were higher for older people, as expected (table 4). However, this was only statistically significant for one component. Component scores also varied as expected by overall rating of the system, with poorer scores associated strongly with a poorer overall rating of the system (table 4). The distribution of component scores across the overall rating of the system suggested that a change of around 0.3 in a component score would indicate a ‘clinically significant’ change.
Component scores for system users by age and overall satisfaction
Discussion
Our preliminary testing of the UCSQ showed evidence of its acceptability, validity and reliability. The response rate of 51% for our postal questionnaire was similar to that of other recent surveys of users of general practice out-of-hours services (46%),6 8 and a recent national general practice survey of five million adults (43%).17 In fact, we obtained a good response rate, given that the questionnaire was sent mainly to people to whom it was irrelevant—that is, non-users of the system. Nonetheless, there is likely to be a non-response bias associated with this response rate. In contrast, the response rate for the market research telephone survey appeared to be poor. However, this is not an unusual response rate for quota sampling using random digit dialling in telephone surveys, where responses of 7% and 9% have been obtained.11 12 Our testing of non-response bias for the two modes of administration as part of a wider study of developing survey methodology for measuring the patient perspective of the emergency and urgent care system identified the telephone survey as superior in terms of representing the age and sex profile of the population—as expected—and also representing the minority ethnic population profile and the reported use of healthcare in a 3-month period.18 The telephone market research approach was also more feasible in the context of routine measurement by system managers.18 In terms of validity, the questionnaire was based on qualitative research with users and tested with users in its early development. There was strong evidence of three constructs which behaved as expected in relation to age and overall satisfaction.
Limitations
We combined two sets of data for the factor analysis—from the postal survey and the telephone survey. We undertook factor analysis on the telephone data only, and then on the postal data only, to see if similar factors were found in each separately. The factors from the telephone dataset were the same as the factors from the combined dataset. The factors of entry and patient convenience were the same for the postal dataset as the combined dataset. However, the progress factor was divided into two separate factors in the postal dataset. There was no conceptual difference between these two factors. A further limitation is that acceptability of a questionnaire is a complex issue, and we have undertaken limited testing of it.
This preliminary test of the questionnaire is promising, but further development is needed. We did not test the reproducibility of the questionnaire using test–retest methods, or its responsiveness to change. However, we plan to use the questionnaire in systems with large changes planned, both before and after any change, in order to assess responsiveness to change.
Conclusions
We have developed and tested a questionnaire which can measure the patient perspective of the emergency and urgent care system—the UCSQ. Future research should address its responsiveness to change.
Acknowledgments
Many thanks to: M Johnson and M Jane at the University of Sheffield for data collection; 2020 market research company for undertaking the telephone interviews; all staff at services and general practices who assisted with administering the surveys; the Urgent Care Network Board for working with us to test the surveys; and everyone who completed the questionnaires.
Footnotes
Funding This work was undertaken by the Medical Care Research Unit, which is supported by the Department of Health. The views expressed here are those of the authors and not necessarily those of the Department.
Competing interests None.
Ethics approval Ethics approval was provided by the 07/Q2403/69 East Midlands Research Ethics Committee UK.
Provenance and peer review Not commissioned; externally peer reviewed.