Abstract
Background
Communication about priorities and goals improves the value of care for patients with serious illnesses. Resource constraints necessitate targeting interventions to patients who need them most.
Objective
To evaluate the effectiveness of a clinician screening tool to identify patients for a communication intervention.
Design
Prospective cohort study.
Setting
Primary care clinics in Boston, MA.
Participants
Primary care physicians (PCPs) and nurse care coordinators (RNCCs) identified patients at high risk of dying by answering the Surprise Question (SQ): “Would you be surprised if this patient died in the next 2 years?”
Measurements
Performance of the SQ for predicting mortality, measured by the area under receiver operating curve (AUC), sensitivity, specificity, and likelihood ratios.
Results
Sensitivity of PCP response to the SQ at 2 years was 79.4% and specificity 68.6%; for RNCCs, sensitivity was 52.6% and specificity 80.6%. In univariate regression, the odds of 2-year mortality for patients identified as high risk by PCPs were 8.4 times higher than those predicted to be at low risk (95% CI 5.7–12.4, AUC 0.74) and 4.6 for RNCCs (3.4–6.2, AUC 0.67). In multivariate analysis, both PCP and RNCC prediction of high risk of death remained associated with the odds of 2-year mortality.
Limitations
This study was conducted in the context of a high-risk care management program, including an initial screening process and training, both of which affect the generalizability of the results.
Conclusion
When used in combination with a high-risk algorithm, the 2-year version of the SQ captured the majority of patients who died, demonstrating better than expected performance as a screening tool for a serious illness communication intervention in a heterogeneous primary care population.
Similar content being viewed by others
INTRODUCTION
Despite ongoing efforts to improve quality of care for patients with serious illness approaching the end of life,1 many patients still receive care that is inconsistent with their preferences, uncomfortable, and costly.2,3,4,5,6,7 Access to palliative care for seriously ill patients is associated with improved patient and caregiver experience and improved outcomes.3,4,5,6,7,8,9,10,11,12,13,14 Nevertheless, use of palliative care interventions, such as those targeting improved communication, remains uneven and suboptimal.7, 15, 16
To improve the value of care for the seriously ill, experts and policy makers have called for a more thorough integration of palliative care into primary care.7, 17, 18 By virtue of their longitudinal relationship with patients and their role in care coordination and access, primary care clinicians are uniquely positioned to support improvements in end-of-life care directly, via delivery of primary palliative care.7, 17, 19 However, the primary care setting is inherently time- and resource-constrained, and access to community resources and specialty palliative care is limited.20, 21 The success of primary care–based palliative care programs, therefore, rests on the ability to focus interventions on the patients who need them most.22 Successful identification of target populations is difficult; primary care populations are heterogeneous and comprised of patients with multiple comorbid illnesses at varying stages of severity. Furthermore, mortality in the primary care population is a relatively rare event,23 complicating efforts to effectively identify patients with poor prognosis.
The “Surprise Question” (SQ)—“Would you be surprised if this patient died in the next 12 months?”24—has shown promise in several populations of seriously ill patients, including those with cancer and renal failure, to identify patients nearing the end of life who might benefit from palliative care interventions.25,26,27,28,29 The SQ is an attractive, simple alternative to mortality predictors that rely on administrative data, which can have poor performance in undifferentiated populations.30 Despite optimism about the potential for integrating the SQ into screening efforts in primary care,24 initial evidence in primary care demonstrated that the SQ may perform poorly.31 Two recently published meta-analyses demonstrate poor to moderate performance of the SQ, finding poorer performance in diseases other than cancer; both call for ongoing research into better means of using the SQ in clinical practice.32, 33
We have been implementing the Serious Illness Care Program (SICP), a multi-step systematic intervention aimed at driving more, better, and earlier communication about goals and values in advancing serious illness, in the primary care setting for the past 4 years.34, 35 Based on discussions about prognostic uncertainty with practicing primary care physicians (PCPs) and the poor performance of the traditional SQ (1 year) in our setting,31 we hypothesized that modifying the horizon of the SQ to 2 years would improve performance. In this manuscript, we present data collected prospectively to evaluate the utility of the 2-year SQ as a tool for prioritizing patients for an early serious illness communication intervention in a heterogeneous, high-risk primary care population.
METHODS
Study Cohort
The study cohort consisted of 1699 chronically ill, complex primary care patients eligible for the Brigham and Women’s Hospital (BWH) Integrated Care Management Program (iCMP) who were screened for the SICP. The iCMP is a primary care–based care management program designed to improve care for complex patients at risk of poor outcomes and high costs.36, 37 Primary care patients are screened for eligibility for the program using a proprietary claims–based algorithm weighing comorbid illnesses and past utilization patterns (https://www.optum.com/solutions/data-analytics/analytics-technology/impact-pro-cpl.html).36 PCPs then review lists of eligible patients and select the most appropriate patients to enroll in the iCMP.36 Then, to identify which iCMP patients were eligible for the SICP, PCPs and nurse care coordinators (RNCCs), answered the 2-year SQ between March 2014 and March 2015.
We asked the SQ via a voluntary electronic survey to both PCPs and RNCCs separately as part of the SICP activities before training and clinicians were aware of the purpose of the survey. The survey had a list of patients for each clinician who were active in the iCMP program and asked the 2-year SQ. A “No” answer to the question represents a clinician concern for death in the next 2 years and thus indicates that a patient may be a good candidate for a serious illness conversation. Study data were collected and managed using Research Electronic Data Capture (REDCap).38 REDCap is a secure, web-based application designed to support data capture, providing (1) an intuitive interface for validated data entry; (2) audit trails for tracking data procedures; (3) automated export procedures for data downloads to statistical packages; and (4) procedures for importing data from external sources.
Outcomes and Covariates
We ascertained the primary outcome, vital status (alive/deceased) 2 years after SQ response, using data from medical records, insurance eligibility files, and the Massachusetts state registry of vital records and statistics; data from the state registry was treated as the primary source unless it was missing, in which case we used insurance eligibility or electronic health record (EHR) data. Secondary outcomes included vital status at 1 year for PCP SQ response and 1 and 2 years for RNCC SQ response. Patient demographic variables and comorbidity data are derived from EHR data and intake surveys completed by nurse care coordinators. Clinicians were surveyed for years in practice, gender, and time spent on clinical duties. We assessed comorbidity (both individual and aggregate burden) using the Gagne methodology, a composite comorbidity index combining Elixhauser and Charlson scores,39, 40 which has been validated for mortality prediction (Appendix).30
Statistical Analysis
For all analyses, we separated the cohort into 2 groups according to the response to the SQ (yes/no). We assessed the performance of the SQ as a predictor of 1- and 2-year mortality stratifying by clinician type. First, a univariate logistic regression model quantified the stand-alone effect of the 2-year SQ response on 1- and 2-year mortality. Next, we incorporated the 2-year SQ into a multivariate logistic regression of 1- and 2-year mortality on additional variables: age, gender, ethnicity, marital status, and comorbidity score. We calculated screening test characteristics—sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive and negative likelihood ratios (+ and −LR respectively), and area under receiver operating curve (AUC)—for physicians and nurses, and constructed collaborative responses that combine responses pertaining to the same patient from both professions.41 Finally, we compared the survival between the SQ response groups after the date when PCP or RNCC screening occurred using Kaplan-Meier curves. We performed all analyses in STATA (version 15.0; Stata Corporation), considering p < 0.05 statistically significant. The institutional review board of Partners HealthCare approved this study.
RESULTS
Of 66 PCPs (58.4%) who returned the survey, 56.1% were female, averaging 16.9 years in practice (interquartile range (IQR) 9–20, 5 missing values), 72.7% completed the training program before returning the survey, and the mean time spent on clinical duties was 68.3% (IQR 32.5–100%, 6 missing values). All 16 nurses answered the survey, 93.8% were female, averaging 23.8 years in practice (IQR 20–30), 68.8% completed the training program, and the mean clinical time was 66.2% (IQR 50–70%, 3 missing values).
For the 1163 patients in this high-risk population for whom their PCP answered the SQ, 57.4% were female and the mean age (recorded at the time of the SQ answer) was 70.1. The most commonly observed Gagne comorbidities in the PCP cohort were hypertension (66.8%), cardiac arrhythmias (27.6%), and chronic pulmonary disease (21.7%). Slightly fewer than half of patients (41.0%) had 3 or more Gagne comorbidities. Most patients were independent for all activities of daily living (60.9%) and either lived with family/spouse (43.3%) or alone (20.6%). Approximately one-third of patients (31.5%) rated their health as fair or poor, and few (11.3%) reported falling in the last 6 months (Table 1). The 1- and 2-year mortality for patients in the PCP cohort was 7.8% and 15.5% respectively. For the 1448 patients in this study for whom their RNCC answered the SQ, 60.4% were female and the mean age was 69.8. The most commonly observed comorbidities in the RNCC cohort were hypertension (68.4%), cardiac arrhythmias (27.6%), and chronic pulmonary disease (22.3%). Fewer than half of patients (41.7%) had 3 or more comorbidities. Functional status, living situation, and self-reported health and falls were similar in the nursing and physician cohorts (Table 1). The 1- and 2-year mortality for patients in the RNCC cohort was 7.1% and 14.7% respectively.
Descriptive Results of Answer to 2-Year SQ
Overall, PCPs classified 452 (38.9%) of these high-risk patients into the “No” group, representing a physician concern for death in the next 2 years, and 711 (61.1%) into the “Yes” group, representing low concern for death in the next 2 years. Among patients whom PCPs identified as at risk of death, 143 patients died in 2 years (31.6% mortality rate); among those whom PCPs assessed at low risk of death, 37 patients died in 2 years (5.2% mortality rate) (Table 2). The 2-year sensitivity of the PCP SQ response was 79.4%, specificity 68.6%, PPV 31.6%, NPV 94.8%, +LR 2.53, and −LR 0.30. In a sensitivity analysis evaluating whether performance changes for the 2-year SQ were related to longer follow-up time and to compare with prior studies examining SQ performance at 1 year, we assessed the performance of the 2-year SQ response for 1-year mortality. The 1-year sensitivity of the PCP response to the 2-year SQ was 84.6%, specificity 65.0%, PPV 17.0%, NPV 98.0%, +LR 2.42, and −LR 0.24.
Nurses classified 352 (24.3%) patients into the “No” group, representing an RNCC concern for death in the next 2 years, and 1096 (75.7%) into the “Yes” group, representing low concern for death in the next 2 years. Among patients whom RNCCs identified as at risk of death, 112 patients died in 2 years (31.8% mortality rate); among those whom RNCCs assessed as low risk of death, 101 patients died in 2 years (9.2% mortality rate) (Table 2). The 2-year sensitivity of the RNCC SQ response was 52.6%, specificity 80.6%, PPV 31.8%, NPV 90.8%, +LR 2.71, and −LR 0.59. In a sensitivity analysis assessing RNCC 2-year SQ performance on 1-year mortality, results were similar to those of the 2-year performance. Kaplan-Meier curves for PCP and RNCC 2-year SQ are presented in Figure 1 (log rank test χ2 = 144.1, P < 0.0001 for PCP, χ2 = 118.2, P < 0.0001 for RNCC curves).
For the 1034 patients for whom there was agreement between clinicians on high risk of death status, where both PCP and RNCC answered “No,” the 2-year sensitivity of the SQ response was 50.3%, specificity 86.7%, PPV 40.0%, NPV 90.8%, +LR 3.78, and −LR 0.57. For patients where either clinician answered “No,” the 2-year sensitivity of the SQ response was 82.6%, specificity 62.7%, PPV 28.1%, NPV 95.3%, +LR 2.21, and −LR 0.28 (Table 2).
Univariate Regression Results
In univariate regression for PCP response, the odds of dying within 2 years for high-risk patients (SQ answer of “No”) were 8.4 times higher than those for low-risk patients (SQ answer of “Yes”) (95% CI 5.7–12.4, p < 0.001) with an AUC of 0.74. For PCP response, the odds of dying within 1 year for high-risk patients were 10.2 times higher than those for low-risk patients (95% CI 5.7–18.3, p < 0.001) with an AUC of 0.75. For RNCC response, the odds of dying within 2 years for high-risk patients (SQ answer of “No”) were 4.6 times higher than those for low-risk patients (SQ answer of “Yes”) (95% CI 3.4–6.2, p < 0.001) with an AUC of 0.67 and the odds of dying within 12 months were similar.
In univariate analysis, patients whom both clinicians agreed were at high risk, the odds of dying were 6.6 times higher than those for low-risk patients (where one or both clinicians deemed the patient to be low risk) (95% CI 4.6–9.6, p < 0.001) with an AUC of 0.69. For patients whom either clinician deemed to be at high risk of death (SQ answer of “No”), the odds of dying were 8.0 times higher than those for low-risk patients (95% CI 5.1–12.3, p < 0.001) with an AUC of 0.73.
Multivariate Regression Results
In multivariate analysis, a PCP high-risk designation was more strongly associated with the odds of 2-year mortality than age, sex, marital status, ethnicity, or Gagne comorbidity score with an odds ratio of 3.2 (95% CI 2.1–5.1, p < 0.001) and an AUC of 0.86. The same was true for nursing response with an odds ratio of 1.8 (95% CI 1.3–2.6, p = 0.001) and an AUC of 0.83 (Table 3). Sensitivity regression analysis removing the SQ yielded similar coefficients for covariables, indicating that it may contribute to prediction independently of other measured patient characteristics (Appendix).
DISCUSSION
In this study, we tested a new 2-year SQ as a screening tool for an early serious illness communication intervention in a diverse high-risk primary care population. The new tool was based on primary care clinician input and initial poor performance with the more traditional 1-year question in a primary care population.31 The 2-year question performed relatively well in predicting mortality at 1 and 2 years and thus for identifying patients for serious illness communication interventions. Patients whom the physician identified as high risk—a “No” response to the SQ—were significantly more likely to die within 1 (unadjusted OR of 10.2) and 2 years (unadjusted OR of 8.4). A nurse response of “No” was similarly associated with a higher likelihood of death, indicating that answering clinician screens in palliative care programs, such as the SQ, may also be a role for nurses within busy primary care practices. Prior studies have examined the SQ in different care settings and with multiple lengths of time, demonstrating considerable variability across settings and underlying conditions.32, 33, 42 AUC for the SQ has ranged from 0.51 to 0.82, and our findings of 0.74–0.75 for doctors and 0.67–0.69 for nurses are within that range. However, while data is sparse in primary care, the SQ has generally performed more poorly in general practice populations, where the AUC of the tool has been lower (0.55–0.59).31, 43
In the current study, we observed the highest sensitivity (82.6%) when either clinician gave a “No” answer to the 2-year SQ. At the same time, the rates of false positive results (a “No” answer in a patient who remained alive) were also relatively low, at 2.6 times the rate of true positives (a “No” answer in a patient who died). This indicates that, by having conversations with all patients with a “No” answer, each clinician team would need to have a conversation with approximately 3 to 4 patients; this effort would result in conversations with over 80% of the patients who died in this population. For clinical practice, our results suggest that using a “No” answer from either the doctor or nurse to the 2-year SQ is a reasonable marker for identifying patients for serious illness communication interventions in the high-risk care management primary care setting: it identifies primary care patients who could benefit from earlier goals and values conversations. Furthermore, in the context of improving serious illness communication, the cost of a false positive result for the SQ is time spent talking with a patient about their goals and values. In that same vein, screening tests without perfect sensitivity may be an acceptable first step in the context of busy clinician practices and clinician shortages. While primary care clinicians work in a highly time-pressured environment, our results suggest that the 2-year SQ is specific enough to avoid generating a significant burden of extra work for clinicians. However, we do not believe these results indicate that the SQ is an appropriate tool for all palliative care interventions, some of which have a narrower application, such as hospice referral or completion of Physician Orders for Life-Sustaining Treatment Forms.
Our study did not analyze the full spectrum of possible uses of the SQ. For example, since the SQ captures information not present in demographics or comorbidities, it is possible that analytical methods could use the SQ in combination with more advanced predictive modeling strategies. However, recent work highlights the significant challenges in accurate prediction of death in patients even using advanced analytic methods especially in the context of targeting interventions.44 In addition, given frequent EMR-generated alerts presented to primary care doctors and attendant “alert fatigue,”45 clinician-driven tools, such as the SQ, hold promise as a means to generate some clinician control and “buy-in,” activating movement from a pre-contemplative state to a contemplative state in considering interventions aimed at improving end of life for their seriously ill patients.46
This study has several important limitations. First, the study cohort was identified as a population at high risk of increased cost and utilization, by a proprietary computer algorithm followed by clinician screen for the program, before the SQ was asked of clinicians and thus the data is most applicable to high-risk care management programs. Future studies should assess the performance of the SQ when paired with other predictive variables and models, as well as the additional possible effect on clinician activation. Furthermore, the physicians, but not nurses, who answered the SQ in this study have experience with the screening question as they had used it previously, although in the first study the question was delivered as part of screening for care coordination, and in this one it was delivered to patients already in the care coordination program as part of the SICP. That familiarity with the SQ and differences in delivery mechanisms may have affected our results when compared with our prior evaluation of the 1-year SQ. Additionally, we did not collect the qualitative or quantitative data needed to answer questions about why the tool performs as it does, for example, what clinicians consider when they answer that they would not be surprised if a patient dies, and this should be a component of further research in this area. Lastly, it is important to note that prediction of death is not the only, and possibly not the most important, factor in identifying seriously ill patients for early palliative care interventions.
CONCLUSIONS
In our study, clinician use of a new 2-year SQ identified primary care patients at increased risk of death and successfully captured most of the patients who eventually died over the next 2 years. The 2-year SQ had notably improved performance over the traditional 1-year SQ in this heterogeneous primary care population. These results suggest that a 2-year SQ holds promise for identification of appropriate patients for serious illness communication interventions in the primary care setting and future work should focus on pairing it with appropriate analytical tools and studying the effect of using the 2-year SQ on physician and nurse behavior, such as conducting discussions about care goals with seriously ill patients.
References
Tulsky JA. Improving quality of care for serious illness: findings and recommendations of the Institute of Medicine report on dying in America. JAMA internal medicine. 2015;175(5):840–1. https://doi.org/10.1001/jamainternmed.2014.8425.
Teno JM, Clarridge BR, Casey V, Welch LC, Wetle T, Shield R, et al. Family perspectives on end-of-life care at the last place of care. JAMA : the journal of the American Medical Association. 2004;291(1):88–93. https://doi.org/10.1001/jama.291.1.88.
Wright AA, Keating NL, Balboni TA, Matulonis UA, Block SD, Prigerson HG. Place of death: correlations with quality of life of patients with cancer and predictors of bereaved caregivers’ mental health. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2010;28(29):4457–64. https://doi.org/10.1200/jco.2009.26.3863
Meier DE. Increased access to palliative care and hospice services: opportunities to improve value in health care. The Milbank quarterly. 2011;89(3):343–80. https://doi.org/10.1111/j.1468-0009.2011.00632.x
Bell CL, Somogyi-Zalud E, Masaki KH. Factors associated with congruence between preferred and actual place of death. Journal of pain and symptom management. 2010;39(3):591–604. https://doi.org/10.1016/j.jpainsymman.2009.07.007
Dahlin CM, Kelley JM, Jackson VA, Temel JS. Early palliative care for lung cancer: improving quality of life and increasing survival. International journal of palliative nursing. 2010;16(9):420–3
Institute of Medicine. Dying in America: Improving Quality and Honoring Individual Preferences Near the End of Life. Washington, DC: National Academies Press; 2014.
Miller SC, Mor V, Wu N, Gozalo P, Lapane K. Does receipt of hospice care in nursing homes improve the management of pain at the end of life? Journal of the American Geriatrics Society. 2002;50(3):507–15
Miller SC, Mor V, Teno J. Hospice enrollment and pain assessment and management in nursing homes. Journal of pain and symptom management. 2003;26(3):791–9
Teno JM, Gozalo PL, Lee IC, Kuo S, Spence C, Connor SR, et al. Does hospice improve quality of care for persons dying from dementia?. Journal of the American Geriatrics Society. 2011;59(8):1531–6. https://doi.org/10.1111/j.1532-5415.2011.03505.x
Teno JM, Shu JE, Casarett D, Spence C, Rhodes R, Connor S. Timing of referral to hospice and quality of care: length of stay and bereaved family members’ perceptions of the timing of hospice referral. Journal of pain and symptom management. 2007;34(2):120–5. https://doi.org/10.1016/j.jpainsymman.2007.04.014.
Pyenson B, Connor S, Fitch K, Kinzbrunner B. Medicare cost in matched hospice and non-hospice cohorts. Journal of pain and symptom management. 2004;28(3):200–10. https://doi.org/10.1016/j.jpainsymman.2004.05.003.
Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early palliative care for patients with metastatic non-small-cell lung cancer. The New England journal of medicine. 2010;363(8):733–42. https://doi.org/10.1056/NEJMoa1000678.
Obermeyer Z, Makar M, Abujaber S, Dominici F, Block S, Cutler D. Association between the Medicare hospice benefit and health care utilization and costs for patients with poor-prognosis cancer. JAMA : the journal of the American Medical Association. 2014;312(18):1888–96.
Gazelle G. Understanding hospice--an underutilized option for life’s final chapter. The New England journal of medicine. 2007;357(4):321–4. https://doi.org/10.1056/NEJMp078067.
Teno JM, Gozalo PL, Bynum JP, Leland NE, Miller SC, Morden NE, et al. Change in end-of-life care for Medicare beneficiaries: site of death, place of care, and health care transitions in 2000, 2005, and 2009. JAMA : the journal of the American Medical Association. 2013;309(5):470–7. https://doi.org/10.1001/jama.2012.207624
Quill TE, Abernethy AP. Generalist plus specialist palliative care--creating a more sustainable model. The New England journal of medicine. 2013;368(13):1173–5. https://doi.org/10.1056/NEJMp1215620.
Tolle SW, Back AL, Meier DE. Clinical decisions. End-of-life advance directive. The New England journal of medicine. 2015;372(7):667–70. https://doi.org/10.1056/NEJMclde1411152.
Lakin JR, Block SD, Billings JA, Koritsanszky LA, Cunningham R, Wichmann L, et al. Improving Communication About Serious Illness in Primary Care: A Review. JAMA internal medicine. 2016. https://doi.org/10.1001/jamainternmed.2016.3212.
Lupu D. Estimate of current hospice and palliative medicine physician workforce shortage. Journal of pain and symptom management. 2010;40(6):899–911. https://doi.org/10.1016/j.jpainsymman.2010.07.004.
Lum HD, Sudore RL, Bekelman DB. Advance care planning in the elderly. Med Clin North Am. 2015;99(2):391–403. https://doi.org/10.1016/j.mcna.2014.11.010.
Billings JA, Bernacki R. Strategic targeting of advance care planning interventions: the Goldilocks phenomenon. JAMA internal medicine. 2014;174(4):620–4. https://doi.org/10.1001/jamainternmed.2013.14384.
Block SD, Bernier GM, Crawley LM, Farber S, Kuhl D, Nelson W, et al. Incorporating palliative care into primary care education. National Consensus Conference on Medical Education for Care Near the End of Life. Journal of general internal medicine. 1998;13(11):768–73
Pattison M, Romer AL. Improving Care Through the End of Life: launching a primary care clinic-based program. Journal of palliative medicine. 2001;4(2):249–54
Pang WF, Kwan BC, Chow KM, Leung CB, Li PK, Szeto CC. Predicting 12-month mortality for peritoneal dialysis patients using the “surprise” question. Peritoneal dialysis international : journal of the International Society for Peritoneal Dialysis. 2013;33(1):60–6. https://doi.org/10.3747/pdi.2011.00204
Cohen LM, Ruthazer R, Moss AH, Germain MJ. Predicting six-month mortality for patients who are on maintenance hemodialysis. Clinical journal of the American Society of Nephrology : CJASN. 2010;5(1):72–9. https://doi.org/10.2215/cjn.03860609
Moss AH, Lunney JR, Culp S, Auber M, Kurian S, Rogers J, et al. Prognostic significance of the “surprise” question in cancer patients. Journal of palliative medicine. 2010;13(7):837–40. https://doi.org/10.1089/jpm.2010.0018.
Moss AH, Ganjoo J, Sharma S, Gansor J, Senft S, Weaner B, et al. Utility of the “surprise” question to identify dialysis patients with high mortality. Clinical journal of the American Society of Nephrology : CJASN. 2008;3(5):1379–84. https://doi.org/10.2215/cjn.00940208
Moroni M, Zocchi D, Bolognesi D, Abernethy A, Rondelli R, Savorani G, et al. The ‘surprise’ question in advanced cancer patients: A prospective study among general practitioners. Palliative medicine. 2014;28(7):959–64. https://doi.org/10.1177/0269216314526273.
Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA : the journal of the American Medical Association. 2012;307(2):182–92. https://doi.org/10.1001/jama.2011.1966
Lakin JR, Robinson MG, Bernacki RE, Powers BW, Block SD, Cunningham R, et al. Estimating 1-Year Mortality for High-Risk Primary Care Patients Using the “Surprise” Question. JAMA internal medicine. 2016. https://doi.org/10.1001/jamainternmed.2016.5928.
White N, Kupeli N, Vickerstaff V, Stone P. How accurate is the ‘Surprise Question’ at identifying patients at the end of life? A systematic review and meta-analysis. BMC medicine. 2017;15(1):139. https://doi.org/10.1186/s12916-017-0907-4
Downar J, Goldman R, Pinto R, Englesakis M, Adhikari NK. The “surprise question” for predicting death in seriously ill patients: a systematic review and meta-analysis. CMAJ : Canadian Medical Association journal = journal de l’Association medicale canadienne. 2017;189(13):E484-e93. https://doi.org/10.1503/cmaj.160775
Bernacki R, Hutchings M, Vick J, Smith G, Paladino J, Lipsitz S, et al. Development of the Serious Illness Care Program: a randomised controlled trial of a palliative care communication intervention. BMJ open. 2015;5(10):e009032. https://doi.org/10.1136/bmjopen-2015-009032
Lakin JR, Koritsanszky LA, Cunningham R, Maloney FL, Neal BJ, Paladino J, et al. A Systematic Intervention To Improve Serious Illness Communication In Primary Care. Health affairs (Project Hope). 2017;36(7):1258–64. https://doi.org/10.1377/hlthaff.2017.0219.
Haime V, Hong C, Mandel L, Mohta N, Iezzoni LI, Ferris TG, et al. Clinician considerations when selecting high-risk patients for care management. The American journal of managed care. 2015;21(10):e576–82
Hsu J, Price M, Vogeli C, Brand R, Chernew ME, Chaguturu SK, et al. Bending The Spending Curve By Altering Care Delivery Patterns: The Role Of Care Management Within A Pioneer ACO. Health affairs (Project Hope). 2017;36(5):876–84. https://doi.org/10.1377/hlthaff.2016.0922
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of biomedical informatics. 2009;42(2):377–81. https://doi.org/10.1016/j.jbi.2008.08.010
Gagne JJ, Glynn RJ, Avorn J, Levin R, Schneeweiss S. A combined comorbidity score predicted mortality in elderly patients better than existing scores. Journal of clinical epidemiology. 2011;64(7):749–59. https://doi.org/10.1016/j.jclinepi.2010.10.004
Sun JW, Rogers JR, Her Q, Welch EC, Panozzo CA, Toh S, et al. Adaptation and Validation of the Combined Comorbidity Score for ICD-10-CM. Medical care. 2017;55(12):1046–51. https://doi.org/10.1097/mlr.0000000000000824.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. https://doi.org/10.1148/radiology.143.1.7063747.
Gomez-Batiste X, Martinez-Munoz M, Blay C, Amblas J, Vila L, Costa X, et al. Utility of the NECPAL CCOMS-ICO((c)) tool and the Surprise Question as screening tools for early palliative care and to predict mortality in patients with advanced chronic conditions: A cohort study. Palliative medicine. 2017;31(8):754–63. https://doi.org/10.1177/0269216316676647.
Barnes S, Gott M, Payne S, Parker C, Seamark D, Gariballa S, et al. Predicting mortality among a general practice-based sample of older people with heart failure. Chronic Illn. 2008;4(1):5–12. https://doi.org/10.1177/1742395307083783.
Einav L, Finkelstein A, Mullainathan S, Obermeyer Z. Predictive modeling of U.S. health care spending in late life. Science (New York, NY). 2018;360(6396):1462–5. https://doi.org/10.1126/science.aar5045
Murphy DR, Meyer AN, Russo E, Sittig DF, Wei L, Singh H. The Burden of Inbox Notifications in Commercial Electronic Health Records. JAMA internal medicine. 2016;176(4):559–60. https://doi.org/10.1001/jamainternmed.2016.0209.
Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot. 1997;12(1):38–48
Acknowledgments
We would like to thank Brandon Neal for his analytical support.
Funding
The implementation of the Serious Illness Care Program at Brigham and Women’s Hospital was funded by Partners HealthCare and the Charina and Branta Foundations.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The institutional review board of Partners HealthCare approved this study.
Conflict of Interest
Dr. Lakin and Dr. Bernacki are supported by the Cambia Sojourns Scholar Leadership Program. Dr. Block serves as the Palliative Care editor for UpToDate. The remainder of the authors declare no conflicts of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Prior Presentations
None.
Rights and permissions
About this article
Cite this article
Lakin, J.R., Robinson, M.G., Obermeyer, Z. et al. Prioritizing Primary Care Patients for a Communication Intervention Using the “Surprise Question”: a Prospective Cohort Study. J GEN INTERN MED 34, 1467–1474 (2019). https://doi.org/10.1007/s11606-019-05094-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11606-019-05094-4