Article Text
PDF
Abstract
Patient satisfaction surveys are an increasingly common element of efforts to evaluate the quality of healthcare. Many patient satisfaction surveys in low/middle-income countries frame statements positively and invite patients to agree or disagree, so that positive responses may reflect either true satisfaction or bias induced by the positive framing. In an experiment with more than 2200 patients in Nigeria, we distinguish between actual satisfaction and survey biases. Patients randomly assigned to receive negatively framed statements expressed significantly lower levels of satisfaction (87%) than patients receiving the standard positively framed statements (95%—p<0.001). Depending on the question, the effect is as high as a 19 percentage point drop (p<0.001). Thus, high reported patient satisfaction likely overstates the quality of health services. Providers and policymakers wishing to gauge the quality of care will need to avoid framing that induces bias and to complement patient satisfaction measures with more objective measures of quality.
- health
- patient satisfaction
- quality of care
- measurement
- Nigeria
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Summary box
Patient satisfaction responses tend to be overwhelmingly positive.
Patient satisfaction ratings are easy to manipulate with the framing of the questions.
The most standard framing of patient satisfaction questions in low/middle-income country contexts—inviting patients to agree or disagree with positive statements—overstates the quality of health services.
Providers and policymakers will need creative strategies to gauge actual patient satisfaction.
Providing a mix of positively and negatively framed statements would attenuate the overall bias, although bias would still be present in the responses to each individual statement.
Avoiding agree/disagree, yes/no response formats would also reduce acquiescence bias.
For comprehensive measures of quality, patient satisfaction must be supplemented with more objective measures.
Introduction
As access to at least some level of health services increases in low/middle-income countries, the focus of policymakers shifts to quality: How can we ensure that patients receive high-quality care? But even while measuring the provision of care is challenging in systems with limited data, measuring the quality of care invites a host of new complications. How can we regularly, systematically measure the quality of medical attention and advice? The simplest, most direct approach seems to be to ask the patients themselves. To gauge the quality of care, many policymakers and researchers turn to the patient satisfaction survey.
In high-income countries, results from patient satisfaction surveys are used to identify gaps and to inform quality improvement plans in healthcare organisations and health systems,1 as well as in research.2 3 Moreover, patient satisfaction is often used as a performance indicator that influences hospital reimbursements and, more and more frequently, physician compensation.4 5 In low/middle-income countries, these surveys are increasingly used. For example, in Africa alone, patient satisfaction instruments have been used in Kenya,6 South Africa,7 8 Nigeria9 and Tanzania,10 11 among others.
These surveys often provide patients with a statement and then ask them to agree or disagree with that statement, such as ‘This health facility is clean. Do you agree or disagree?’ If patients answer these questions favourably, does that actually reflect high levels of patient satisfaction, or rather does it reflect a bias? Patients in low-income environments with few options for health services may value any services, and indeed, other work indicates high reported patient satisfaction even in the face of relatively low-quality services.12 Alternatively, patients may agree with the interviewer to be agreeable (‘acquiescence bias’), or because ‘I agree’ is the first option offered and so choosing it requires the least cognitive effort (‘satisficing’).13
This is a substantive issue: of 26 recent patient satisfaction surveys in low/middle-income countries, more than three-quarters phrased their statements positively. Specifically, of the 26 studies included in the World Bank’s Central Microdata Catalog that used patient satisfaction questions,14 20 (77%) were phrased positively and only six (23%) were phrased negatively or neutrally. This potential framing bias adds to other previously identified challenges with patient satisfaction surveys, such as that patient satisfaction measured at clinics is rated much higher than patient satisfaction measured at home.15
How can we distinguish true patient satisfaction from bias induced by the survey?
We implemented an experiment in Nigeria to distinguish between actual satisfaction with health services and survey biases. The study was implemented in 80 primary healthcare centres in six Nigerian states: Anambra, Bauchi, Cross River, Ekiti, Kebbi and Niger. Patient exit interviews were administered to all patients who visited the primary healthcare centres at the time of data collection. Surveys were administered face-to-face by trained enumerators with tablet computers in eight monthly rounds between June 2014 and February 2015. Interviewers arrived unannounced as part of a larger randomised controlled trial that involved helping clinic staff to identify gaps in the quality of service delivery and to set goals to close those gaps.16 The patient exit interview did not mention the larger quality improvement intervention. In total, 2222 patients were interviewed, or roughly 28 patients per facility on average. In addition to patient satisfaction measures, data were collected on a set of demographic and socioeconomic characteristics of the patients (including age, gender, education, employment and income).
Each patient was presented with 11 statements on the quality of care and asked to agree or disagree with each statement. Patients were randomly assigned to receive one of three treatments: the standard, positively framed statements (table 1 Set A), a set of equivalent negatively framed statements (table 1 Set B), or a random mix of the two. As expected with randomisation and a large sample of patients, patients were statistically indistinguishable across groups on age, gender, education and employment.
Positive and negative framed patient satisfaction statements
How patient satisfaction questions are framed makes a big difference
With positively framed statements, patients report extremely high levels of satisfaction. There is no item for which approval is lower than 88%; for more than half of the statements, agreement exceeds 94%. However, when patients are presented with negatively framed questions, satisfaction drops significantly on 10 out of 11 questions, with an average drop of 7.5 percentage points across all questions and including drops as large as 18.9 and 11.6 percentage points (figure 1 and table 2). (See online supplementary appendix for details on the specification and the robustness analysis.)
Supplementary file 1
Patient satisfaction is easily manipulated by framing of questions. Percentage of patients who respond ‘I agree’ to a statement about the quality of care that they received at a primary healthcare facility. Based on a sample of 2222 patients across six Nigerian states.
Impact of framing on patient satisfaction—the simple specification
When we separate the results for those who received positive and negative statements within a mixed battery of statements from those who received purely positive or purely negative statements, the pattern remains the same (table 3). For 10 out of the 11 statements (and on average), the impact of negative statements with all negative statements is negative and statistically significant. Patients who received a negative statement in the mixed battery of statements were also less likely to respond favourably. Here, 8 of the 11 statements show significant effects.
Impact of framing on patient satisfaction—the detailed specification
To probe the robustness of the results, we explore whether these results are consistent when we control for patient gender, age, income levels, or education levels. In all cases we obtain very similar results. We also introduce interaction terms for these variables, for which we do not see systematically significant results. That is, the pattern of acquiescence bias that we uncovered seems to affect patients irrespective of their income, education and experience with private facilities.
These results suggest that high-reported patient satisfaction likely overstates the quality of health service provision in resource-constrained environments, adding to evidence that patient satisfaction is imperfectly related to health outcomes.4 Inflated patient satisfaction reports can potentially distort decisions about effort and resource allocation.
Conclusion
There is broad consensus that improving patients’ experience as they obtain healthcare is an intrinsically desirable goal. Some elements of that improved experience are likely to be universal: patients value short waiting times and clean facilities, and they appreciate providers that respond to their needs and treat them with respect. Other elements may vary across contexts, such as the extent to which patients value being involved in the medical decision-making process. Routine measurement of patient experience and satisfaction is becoming commonplace in healthcare organisations in both high-income countries and low/middle-income countries.
Our results demonstrate that patient satisfaction measurements are deeply sensitive to the framing of the questions. Specifically, we find strong evidence of acquiescence bias, or the tendency of individuals to agree to the statement they are presented, irrespective of its content. As such, the standard (‘positive’) formulation results in consistently inflated measures of patient satisfaction. This highlights the need to supplement patient satisfaction with other measures to provide an overall indication of service quality. These may include the measurement of actual health outcomes, as well as the use of vignettes to gauge provider knowledge and standardised patients to gauge provider effort. Furthermore, there may be significant ceiling effects with positively framed questions, since the average tends to be so high that it is difficult to distinguish across performance levels.17 18
The main implication of our study is that designers of patient satisfaction surveys should avoid using all positively phrased statements. Providing a mix of positively and negatively framed statements would attenuate the overall bias, although bias would still be present in the responses to each individual statement. Avoiding agree/disagree, yes/no response formats would also reduce acquiescence bias. Several major patient satisfaction surveys in use already incorporate these recommendations. For example, the 15-item Picker Patient Experience Questionnaire avoids agree/disagree statements altogether,19 and the Patient Experience Questionnaire has agree/disagree statements but includes both positive and negative framing.20
Reduced bias would make patient satisfaction measures more meaningful, allowing better distinguishing across facilities, and would be beneficial for programmes wishing to use patient satisfaction to identify gaps and areas where changes are needed.
Acknowledgments
The authors thank the Nigerian Federal Ministry of Health and the Nigerian National Primary Healthcare Development Agency (formerly known as the Saving One Million Lives initiative) for support, the Bill and Melinda Gates Foundation for funding, Ezinne Eze-Ajoku and Kaveh Shojania for input, and Anna Popova and Fei Yuan for research assistance.
References
Footnotes
Handling editor Seye Abimbola
Contributors All authors contributed equally to this manuscript.
Funding The data collection for this research was funded through a grant by the Bill and Melinda Gates Foundation.
Competing interests None declared.
Patient consent For this non-invasive survey of patient satisfaction, the IRB allowed verbal consent.
Ethics approval This research was approved by the Institutional Review Boards (IRB) of the Government of Nigeria and Johns Hopkins University (HIRB 00001960).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All data from the study are available upon request from the authors.