Article Text
Abstract
Objectives Oncologists need competence in clinical prognostication to deliver appropriate care to patients with cancer. Most studies on prognostication have been restricted to patients in palliative care settings. This paper investigates (1) the prognostic accuracy of physicians regarding a broad cohort of patients with cancer with a median life expectancy of >2 years and (2) whether a prognosis training can improve prognostication.
Design Prospective single-centre study comprising 3 phases, each lasting 1 month.
Setting Large teaching hospital, department of oncology and haematology, Germany.
Participants 18 physicians with a professional experience from entry level to 34 years. 736 patients with oncological and malignant haematological diseases.
Interventions Baseline prognostication abilities were recorded during an ‘untrained’ phase 1. As an intervention, a specific prognosis-training programme was implemented prior to phases 2 and 3. In phase 3, physicians had to provide additional estimates with the inclusion of electronic prognostic tools.
Outcome measures Prognostic estimates (PE) were collected using ‘standard’ surprise question (SQ), ‘probabilistic’ SQ (both for short-term prognostication up to 6 months) and clinician prediction of survival (CPS) (for long-term prognostication). Estimated prognoses were compared with observed survival. Phase 1 was compared with phases 2 and 3.
Results We included 2427 PE for SQ, 1506 for CPS and 800 for probabilistic SQ. Median OS was 2.5 years. SQ accuracy improved significantly (p<0.001) from 72.6% in phase 1 to 84.3% in phase 3. Probabilistic SQ in phase 3 showed 83.1% accuracy. CPS accuracy was 25.9% and could not be significantly improved. (Electronic) prognostic tools—used alone—performed significantly worse (p<0.0005) than physicians and—used by the clinicians—did not improve their performance.
Conclusion A specific prognosis-training programme could improve short-term and intermediate-term prognostication. Improvement of long-term prognostication was not possible. Inexperienced residents as well as experienced oncologists benefited from training.
- oncology
- education & training (see medical education & training)
- prognosis
Data availability statement
Data are available on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This is the first study that presents a structured training programme for oncologists that has a positive impact on clinical prognostication.
This prospective study included a large cohort of patients with different oncological and malignant haematological diseases at all disease stages.
Prospective follow-up data of almost 4 years were collected, which allowed for the assessment of longer prognosis estimates.
The effect of training was not demonstrated by a randomised comparison but by comparing baseline (untrained) prognostication abilities with the performance after training in the same group of physicians.
The study is a single-centre study—even though performed at a large teaching hospital.
Introduction
In order to deliver appropriate care to patients with cancer, an oncologist needs competence not only as a diagnostician or as a therapist but also as a prognosticator. The relevance of prognostication may not always be obvious, but it underlies virtually every aspect of patient care. For example, the same clinical symptom (severe respiratory insufficiency) may lead to very different courses of action (eg, transfer to the ICU vs best supportive care measures) depending on the estimate of the patient’s prognosis by the treating physician. The oncologist may have made these estimates consciously or unconsciously, but they nevertheless constitute the background in front of which everyday patient care takes place.1
Prognostication is an uncertain business and the ability of oncologists to estimate the future course of their patients is limited. Most explicit analyses of physicians’ abilities to prognosticate have been made in the palliative care setting on patients with very short life expectancies.2–29 In contrast, this paper evaluates the prognostic abilities of residents and experienced oncologists in a department of oncology and haematology in a major acute care teaching hospital, where patients had an average life expectancy of >2 years. This paper also investigates how far a training programme can improve prognostic performance.
There are three major approaches to indicate the prognosis of a patient: (1) the temporal approach (also called ‘clinician prediction of survival’ (CPS)), (2) the surprise question (SQ) and (3) the probabilistic approach. Of these, the temporal approach is the most commonly used method within the literature.29–32
When using the temporal approach, the prognosticator is required to state a specific period in absolute numbers (days, months, years). Many studies consider the prognosis to be ‘accurate’ if the CPS is within the range 67%–133% in relation to the actual survival of the observed patient (which is set as 100%).6 7 11 13 15 16 18 21 The SQ, which was originally developed as a screening tool for the initiation of palliative care, requires physicians to answer whether they would be surprised if their patient had died within a certain time. There are only two options (yes/no) and the estimation period is defined (eg, 6 months). The SQ is accurate if the answer is ‘yes’ and the patient has survived or if the option is ‘no’ and the patient has died. The third option is the probabilistic approach. Here, the respondent must indicate the probability that a patient has or has not died at a given time. (Like in the SQ) the period of interest is defined (eg, 6 months), but the prognostic estimate is provided in percentage increments (usually 10%) and thus has a quantitative component.
Prognostication per se (= estimating a prognosis) is not the same as conveying a prognosis to a patient (eg, ‘breaking bad news’). The latter ability (which is not the topic of this paper) is also of high relevance and is generally regarded to require considerable experience and clinical expertise (which is considered teachable). In contrast, the preceding process of estimating the prognosis—that is, prognostication as such—is described as ‘Medicine’s Lost Art’ and remains a black box in two ways: (1) The prognostic performance of clinicians—outside the inpatient palliative care setting—is often unknown. (2) Training of prognostication is often ‘implicit’—if it happens at all.33–35 This is in contrast to other fields outside clinical medicine where systematic training approaches to increase prognostic performance and accuracy do exist.36 37
For patients in routine oncological practice (and not in the palliative care setting)—no data on improving prognostication via a training programme exist to our knowledge. In this paper, we, therefore, evaluate not only the prognostic performance of residents and experienced oncologists in the acute care setting but also how far a structured training programme might have a positive impact on prognostication.
Methods
Time period and general conditions
The prospective study was conducted at the clinic for oncology and haematology at the Hospital Barmherzige Brüder in Regensburg, Germany. It consisted of 3 different phases with a duration of 1 month each, taking place in the 3 months of April, June and September of 2019. Data collection was performed as part of routine care on the oncology wards and in the oncology outpatient clinic. All physicians agreed to participate in the study and gave their informed consent. Since the study did not involve any intervention affecting the patient and the standard of care was not affected no informed consent was required for patients. The current study was carried out in accordance with the Declaration of Helsinki.
Participants
The participants of the study consisted of the whole medical team (= physicians) of the department of oncology and haematology (n=21), who (individually) generated the prognostic estimates on the patients (described below). This ‘whole group’ was subdivided into ‘residents’ (physicians in training, less than 1 year of training in clinical oncology = ‘inexperienced’) and into ‘oncologists’ (board-certified oncologists with a minimum experience in clinical oncology of 13+years = ‘experienced’).
Procedure
General procedure
Patients who were discharged from the department of oncology and haematology during the 3 phases were included. All patients were required to have a malignant disease but could be in any stage of their disease. After each patient’s discharge, the resident physician who last cared for the patient and the supervising senior physician and/or chief physician independently evaluated the patient’s prognosis based on knowing the patient personally and based on the information from the discharge letter. To determine both a qualitative value and a quantitative value for the prognostic accuracy, questionnaires were used to elicit the physicians' responses to the following two questions:
‘Would you be surprised, if your patient had died in 6 months from today?’ (‘surprise question’ (SQ)).
‘If you were faced with a large group of patients with a very similar constellation of age, disease, comorbidities, prior therapies as in the patient whose discharge letter you have just completed, what median survival would you expect in this group?’ (‘clinician prediction of survival’ (CPS)).
Sample size calculation/considerations
We determined that each of the 3 phases was to consist of 4 consecutive weeks. Our calculation was that within this period of 4 weeks approximately 300 patients would leave the oncology department and could, therefore, be evaluated for prognosis by their treating physicians. Since the resident as well as one or two senior oncology consultants gave their prognosis estimates (PE) independently on the same patient a total number of approximately 600–900 prognoses per phase were expected (and de facto achieved—phase 1=925, phase 2=712, phase 3=790). We assumed that these rather high numbers (as compared with those within the literature of clinical prognostication) were needed to give an accurate descriptive picture of the status quo on ‘untrained’ clinical prognostication (in phase 1) and of prognostic performance after training (phases 2 and 3).
Data collection
Every physician’s estimate and every hospitalisation of a patient were considered. When there were multiple estimates for a patient (eg, by resident, senior physician and chief physician), all three estimates were recorded as distinct data points for this respective patient for this hospital stay. Patients, who received a PE on more than one occasion, were included in the analysis for each of their PE. For analysis all physicians together formed the ‘whole group’. In addition, according to the clinical experience, we defined the subgroups ‘residents’ and ‘oncologists’ (board-certified specialists in haematology and oncology).
To avoid the possibility of non-representative values caused by outliers, only physicians who provided ≥10 PE per phase were included in the final analysis. Physicians, who did not work in the oncology wards in every phase (eg, due to rotation in residency training), were included only in the involved phase(s) if they had completed ≥10 PE. Real survival data (date of death or last recorded life status) were provided by the Tumor Center Regensburg in March 2023, with a follow-up of approximately 4 years. PE for patients, who did not have an oncological disease or who were not registered at the Tumor Center Regensburg, were not included in the analysis (because follow-up could not be ensured).
Phase 1 versus phases 2 and 3/interventions
A flow chart of the 3 phases with all participating physicians and patients is provided in online supplemental file 1. In phase 1, the PE was performed with no prior training and was thus based mainly on prior clinical experience and ‘gut feeling’ (‘thinking fast’). In contrast, during phase 2 ‘thinking slow’ (according to Kahneman38) was encouraged—that is to consciously take time for the PE. To enable that, a prognosis training was provided for the physicians prior to the start of the second phase and repeated before phase 3 (see online supplemental file 2):
Supplemental material
Supplemental material
The prognosis training consisted of a brochure designed to support the physicians with the aid of epidemiological data on the 21 most common oncological diseases of the clinic. For each disease, general information was provided on the median survival time across all groups as well as information for different patient groups (eg, according to patients’ age, stage or mutation status). Corresponding Kaplan-Meier curves were provided with information on absolute and relative survival for these subgroups. Sources of the information were derived mostly from the Munich Tumor Registry, other sources were, for example, the website UpToDate or clinical or epidemiological studies. These epidemiological data were intended to be used as numerical ‘anchors’ during the (conscious) process of prognostication.
The brochure was introduced to the participating physicians within a teaching lecture during which a general approach to prognostication was provided (see online supplemental files). In brief, this approach assumes that prognostic accuracy can be improved by first taking an ‘outside view’ on a situation—for example, by using anchor numbers derived from the brochure (‘actuarial’ approach).39 In subsequent steps, a more ‘personalised’ and multifaceted ‘inside view’ may follow in which relevant details of the individual patient can be used to adjust the prognostic estimate—using the principle of ‘Fermisation’ (referring to Enrico Fermi—see online supplemental files).
Supplemental material
Supplemental material
Supplemental material
In addition, the SQ of the questionnaire was reworded in phase 2 to exclude the subjective factor of the term ‘surprise’. (‘Do you think it is more likely (> 50%) that your patient will be alive in 6 months than that he will have died by that time?’)
Prior to the start of phase 3, prognostic tools were provided for the most common oncological diseases of the clinic, if they were available either as a website or as an app (see online supplemental file 3). In this phase, physicians had to provide two separate PE on the same patient—the first without knowledge of the PE made by the respective (electronic) prognostic tool and the second while being aware of the result.
In phase 3, the questionnaire was complemented by another version of the SQ—that is, the ‘probabilistic SQ’: In addition to the answer option ‘yes/no’, the SQ was now to be answered probabilistically (‘Please indicate the probability (in 10% increments from 0% to 100%) that your patient will be alive in 6 months.’).
Statistical evaluations
CPS was evaluated using the three categories ‘accurate’, ‘neutral’ and ‘inaccurate’. When the real OS time was set as 100%, we set the definition of ‘accurate’ estimates as those that ranged from 67% to 133% of this value. ‘Neutral’ estimates were in the range of 34%–67% and 133%–166% and ‘inaccurate’ estimates were beyond these limits, that is, <34% and >166% of the actual observed survival time.
The SQ was considered ‘correct’ if the answer was ‘yes’ and the patient was alive at 6 months from the time of the PE or the answer was ‘no’ and the patient had died within 6 months. In the probabilistic form of the SQ, we defined estimates as ‘correct’ if the patient had died at 6 months when given answers ranging from 0% to 40% or lived at 6 months when given answers ranging from 50% to 100%.
The rate of correctly answered SQ and the rate of ‘accurate’ CPS PE of the ‘whole group’ were collected as the primary endpoints of this study.
As a secondary endpoint, we aimed to describe the potential impact of a ‘prognosis training’ on the prognostic performance of physicians. To analyse this intervention, performance in phase 1 (‘untrained’) was compared with the performance after training (both phases 2 and 3 after training—‘trained’) and tested for statistical significance by using a χ2 test. These analyses were performed for the ‘whole group’ and for the two subgroups ‘residents’ and ‘oncologists’ for each of the 3 phases.
In addition, ‘residents’ and ‘oncologists’ were tested against each other for differences in prognostic performance in each phase.
For the probabilistic form of the SQ, we tested how far the estimated survival likelihood correlates with the actual survival (χ2 test for linear trend). The basis of this analysis is the groups of patients who had in common their respective estimated survival likelihoods—that is, ‘0% group’, ‘10% group’, ‘20% group’, etc up to the ‘100% group’. For example, the ‘100% group’ comprises all patients, which had been given a survival likelihood of 100% at 6 months by their respective physicians. In this probabilistic setting we also calculated the Brier score for the overall group and the subgroups (oncologists and residents).
The programme Microsoft Excel was used for descriptive statistics of the 3 phases as well as for documentation, analysis and creation of diagrams. The presentation of Kaplan-Meier curves was realised with the statistical program IBM SPSS Statistics 29.0.
Patient and public involvement
None.
Results
Test persons/assessing physicians
21 physicians participated by providing a total of 2486 prognostic estimates on 748 individual patients. 18 physicians could be included, 3 were excluded because they provided <10 PE.
The whole group was subdivided into two groups. The ‘residents’ group was formed by rather inexperienced physicians and consisted of 11 different residents. The ‘oncologists’ group consisted of seven experienced board-qualified haematologists-oncologists. The professional experience of the residents averaged from entry level to 6 years of experience in internal medicine with a maximum experience in clinical oncology of 1 year. The oncologists had an experience in clinical oncology ranging from 13 to 34 years.
Patients
A total of 748 patients with a wide range of diseases were assessed—12 of which were excluded from the analysis because they either had no oncologic disease or no status on survival was available at the time of follow-up, so the final analysis consisted of 736 patients. The patients’ characteristics can be seen in table 1. Of 736 patients, 410 (55.7%) had died by the time of the last follow-up in March 2023. The median overall survival was 2.5 years (see figure 1A).
Clinician prediction of survival
Comparison of phase 1 with phase 2 and 3
For CPS—that is, ‘long-term’ prognostication—1506 PE were available for evaluation. Overall, only 25.9% of them were ‘accurate’ according to the definition (±33% of actual observed survival). Accuracy varied over real survival as shown in figure 1B—with the best results between 8 and 10 months. Over the 3 phases, there was a numerical trend towards a slightly better accuracy from 23.6% to 28.0%—which was also observed within the two subgroups (residents and oncologists)—which however lacked statistical significance in all subgroups (see table 2).
When Kaplan-Meier curves of real and estimated overall survival were plotted against each other, the curves crossed at about 9 months for the whole group of patients. There is overestimation of OS in the ‘immediate’ future whereas prognosis in the more distant future tends to be underestimated. This finding is also backed by the dot plot (figure 1C) which depicts the estimated survival over real survival. Corresponding analyses of the 3 phases can be found in online supplemental file 4A–C. Up to a real survival of 9 months, we observed a tendency towards overestimation (671/853 (78.7%) above the diagonal of ‘perfect prediction’). After 9 months, there is a tendency towards underestimation (451/653 (69.0%)) (see online supplemental file 4D).
Over time (from phases 1 to 3) the Kaplan-Meier curves tended to cross earlier—potentially indicating more conservative estimates after training. When comparing differences between the ‘estimated’ CPS and the ‘real’ OS curves there is a visual trend towards closer curves after training (phase 2 and 3) as compared with the ‘untrained’ phase 1 in the ‘immediate’ future (see online supplemental file 4). Overestimation of survival—which was very evident in phase 1—seemed to be less pronounced after training (no test for significance).
Surprise question
Option ‘yes’/‘no’
In the evaluation of the SQ with the answer option ‘yes’/’no’, a total of 2427 answers were included. 76.8% of them were correct and 23.2% incorrect. Physicians achieved 72.6% accuracy in phase 1, 74.0% in phase 2 and 84.3% in phase 3. There was a highly significant (p<0.001) improvement in accuracy over time during the 3 phases for the whole group. Residents had an accuracy of 60.7% in phase 1, 63.2% in phase 2 and 81.1% in phase 3, which demonstrated a significant (p<0.01) improvement. The accuracy of the oncologists also increased significantly (p<0.05) from 75.7% in phase 1 to 77.0% in phase 2 to 85.6% in phase 3. In total, oncologists achieved a significantly (p<0.005) higher accuracy (79.1%) as compared with residents (69.5%). However, this was only significant in phase 1 and in phase 2, but not in phase 3 (see table 3).
Probabilistic SQ
800 answers were available for the evaluation of the probabilistic SQ in phase 3 of which 83.1% were correct. The probabilistic SQ showed a high accuracy in the extremes of the spectrum, the answer ‘0%’ and ‘100%’ even showed an accuracy of 100.0% with 27/27 correct SQ (‘0%’) and 53/53 correct SQ (‘100%’). The closer the answer approached the mid-range values, the lower was the accuracy. These mid-range percentages (answers 40%–60%) only had a low accuracy of 54.5% which was significantly (p<0.0001) worse compared with the ‘extreme’ parts of the spectrum (0%–30% and 70%–100%) with a correct estimate in 89.9% of cases. This expression of mental indecisiveness is—of course—the equivalent of a coin flip. A detailed overview of the probabilistic SQ is shown in figure 2A. The values for the Brier score were 0.193 for the whole group, 0.191 for the oncologists and 0.199 for the residents.
An interesting finding for groups of patients was that a predicted higher likelihood of survival (eg, ‘group 90%’ vs ‘group 60%’) also correlated with a higher probability of real survival in these groups in a similar magnitude (p<0.0001) (see figure 2B). Therefore, the ‘coin flip’ estimate is not ‘worthless’ but carries information because the ‘group 50%’ comprises a group of patients of which roughly half will have died after 6 months.
Accuracy of prognostic tools (SQ, CPS)
During phase 3, established (electronic) prognostic tools were available and suitable for only 34.6% of patients and their respective clinical situation. When used alone for the SQ the accuracy of these prognostic tools was only 70.2%, which was significantly worse (p<0.0005) than the performance of the whole group (84.3%) and of the oncologists (85.6%). It was also numerically—if not statistically significant—worse than the performance of residents (81.1%).
When prognostic tools were used for CPS estimates, an ‘accurate’ result was only achieved in 16.9% which was less than the rate of the whole group of physicians 25.9% (and of all subgroups). In a subgroup (n=443 patients) physicians made PE on CPS first without and then with the respective prognostic tool—however, accuracy was not significantly increased (28.0% to 29.4%).
Discussion
This prospective study examined physicians’ accuracy in prognostic estimates (PE) for oncological patients of all stages as well as a potential training effect. To do this, nearly 2500 PE on over 700 oncological patients were performed. We analysed three types of PE in our study: the (standard) SQ, the probabilistic SQ and CPS. The patients in our study had a median survival of 2.5 years, which is substantially longer than in most prior studies on prognostication in patients with cancer which have been performed on patients in very advanced stages or in a palliative setting.2 4–16 18 19 21 25–28
Main findings
In our study, the whole group of participating physicians achieved an accuracy of 76.8% when using the SQ. This result is fairly comparable to the summarised accuracy of 78.6% for oncological patients calculated by a systematic review in 2017.40 It should be noted, however, that very different time intervals have been investigated so far when answering the SQ for oncological patients and that the patient cohorts differed considerably (see online supplemental file 5).25–27 Quite evidently, the time interval chosen for the SQ has an impact on accuracy, since the longer the time interval the less certain the estimates will be and the accuracy will ‘suffer’. The selection of a time interval appropriate for the respective patient population is therefore essential.
Interestingly, the prognostic accuracy of the whole group of participating physicians increased after training from an initial 72.6% to 84.3% in the last phase. This effect was especially pronounced in residents who started from a lower level (60.7% in phase 1) and then achieved 81.1% in the final phase (+20.4%). The experienced oncologists started from a significantly higher level of accuracy (75.7%) but also benefited from training (85.6% in the final phase)—however, to a less pronounced degree in absolute terms (+9.9%).
Our data indicate that experienced physicians are better prognosticators than early career physicians which has so far been a controversial issue within the literature.8 16 21–23 41 42 Moreover, a prognosis-oriented training programme has a potential to improve the prognostic accuracy of both residents as well as of experienced oncologists. Such a focused training programme has been repeatedly recommended in the literature.8 42 43 While some guidelines and training concepts exist in the palliative setting.33–35 This is—to our knowledge—the first study to have implemented and systematically tested such an approach in a large group of oncological patients with a median survival over 2 years. (Our intention was to develop and test a short training programme and find out whether there is an effect. Our aim was not to specifically identify and isolate each reasoning principle within the programme and determine its incremental effectiveness.)
Wider implications
Prognostication using the SQ for a 6-month period can be considered ‘short-term prognostication’. In this situation, physicians’ accuracy was substantially better than chance and it was trainable. This finding contrasted with the situation of ‘long-term prognostication’ in which physicians were required to estimate overall survival (which most of the times exceeded 6 months—as demonstrated by a median OS of 2.5 years). The CPS was mostly ‘off-target’ with only 25.9% ‘accurate’ PE. Our results are in accordance with smaller studies which used the same definition for accuracy and demonstrated accuracy values between 20% and 35%.6 7 11 13 15 16 18 20 21 As mentioned above, previous studies assessed very advanced cancer patients with an OS of only a few months so that survival data were available for nearly all patients.6 7 11 13 15 16 18 21 Since only 60% of our patients had died at the time of analysis further follow-up of our patients might change the numerical value for accuracy of CPS.
CPS and SQ at 6 months are different metrics and, therefore, the absolute values of these parameters cannot be directly compared. However, CPS can be ‘converted’ into the format of an SQ by assuming the following: The groups ‘low inaccurate’, ‘low moderate’ and ‘accurate’ will not have been ‘surprised’ that the patient has died at his/her specific time of death. To illustrate this, imagine the following thought experiment: If we (retrospectively) know that a patient has died 450 days after discharge from our care we can pose this constellation (discharge letter from that past discharge) to a group of physicians in the form of an individual SQ: ‘Would you be surprised if this patient had died within 450 days?’ A physician with a CPS of ‘200 days’ would not be ‘surprised’ as in our SQ definition and he would be ‘right’. In contrast, a physician stating a CPS of ‘700 days’ would be ‘surprised’ by the earlier demise of the patient and his estimate would have been ‘wrong’. This also illustrates that the SQ is the more ‘permissive’ metric because it considers estimates as correct (‘not surprised’) that are not in the ‘accurate’ category of the CPS definition—that is, ‘low inaccurate’ and ‘low moderate’.
However—even when using the more permissive ‘individual SQ’ definition (=consider the sum of ‘low inaccurate’, ‘low moderate’ and ‘accurate’ as ‘not surprised’ and therefore as ‘correct’)—the value for the CPS for the whole group is only 53.8%—hardly better than chance. Interestingly, there is no (significant) trend towards improvement by training and also there seems to be no difference between residents and experienced oncologists. These findings can be used to estimate the ‘horizon’ of individual clinical prognostication—which is a controversial issue in the literature12 17 19 21 22 24 42 43—in our study: The group of patients on which definite survival data were available (ie, had died at the time of analysis) and on which our CPS analysis was performed had a median (50%) survival of 9 months and 60% of patients had died within 1 year (data not shown). Assuming that prognostication over longer periods (than 1 year) will be even worse, we conclude that whereas individual prognostication within a time frame of 6 months by physicians is better than chance (and is trainable) this seems no longer to be the case when a time horizon of circa 1 year has been reached.
It is relevant to differentiate between prognostication for an individual patient and prognostication for a group of patients. During phase 3, when we also analysed the ‘probabilistic SQ’. Its overall accuracy of 83.1% was nearly identical to that of the ‘standard SQ’ in that phase (84.3%) as expected. These are the numerical values that are relevant for the PE of individual patients. However, unlike the standard SQ, the probabilistic SQ also had a quantitative component (‘estimate the probability of survival at 6 months in 10% increments!’). This has two consequences:
The forecasts in this format convey a sense of decisiveness and indecisiveness and thereby contain additional prognostic information/certitude: This is demonstrated by the high accuracy of the extreme ends of the spectrum (0%–30% = likely to die within 6 months; 70%–100% = likely to survive the next 6 months) which reached 89.9%. This was substantially higher than in the middle (‘indecisive’) part of the spectrum (40%–60%) where the accuracy was only 54.4%.
This ‘indecisive’ accuracy of 54.4% in the mid-range is hardly better than chance and implies a ‘coin flip’ for the individual patient. However, this does not mean, that such a prognostic estimate contains no information at all and is basically worthless. In contrast, we could show that the collective prognostic estimates in 10% increments closely mirrored the de facto survival rates of groups at 6 months as demonstrated in figure 2B. For example, the ‘group 50%’ comprises a group of patients (all considered by their treating physician to have a 50% likelihood of survival at 6 months) of which roughly half will in fact die within 6 months.
Even though each physician performed discrete prognostic estimates on an individual patient the sum of all these forecasts nevertheless also contains prognostic meaning for groups. This ‘wisdom of the clinical crowd’ is also exemplified by the survival curve of our patient cohort (figure 1A) which is fitted quite well (though not perfectly) by the multitude of forecasts done on these patients by the participating physicians.
Another aspect that can be derived from these two Kaplan-Meier curves is that overestimation of prognosis tends to occur predominantly in the early phase (6–9 months) whereas underestimation tends to be predominant at the end of the first year of observation and thereafter. The two curves cross between 6 and 9 months after t0 (= time of the prognostic estimate) with the crossing tending to be earlier in the later phases—which may have been due to more conservative forecasts after training. This finding is in accordance with Fairchild et al who also observed an overestimation for patients with a survival of less than 6 months and an underestimation for patients with a survival of more than 9 months.12 We have also visualised this finding in the dot plot (figure 1C) and found a higher rate of overestimation for survival <9 months and a higher rate of underestimation for survival >9 months.
Established prognostic tools were only available for a minority of patients in their specific clinical situation (34%) and—when used alone—performed worse (accuracy 70.2%) than physicians (84.3%) in phase 3. They also did not add to prognostic accuracy of physicians. This might have been due to the fact, that often these tools are optimised for only one specific situation (eg, primary diagnosis and start of curative therapy). In these situations, some of the tools worked very well. However, most tools were not or less helpful in the many other situations when a clinical encounter takes place (eg, 3 months into second-line therapy). Also most of the time these tools convey ‘collective’ prognoses and place a patient into a prognostic group—similar to our results of the probabilistic SQ. This can of course be helpful, but it is a more permissive/easier task than individual prognostication.
Strengths and limitations
This is the first study that presents a structured training programme for oncologists that has a positive impact on clinical prognostication. This prospective study included a large cohort of patients with different oncological and malignant haematological diseases at all disease stages. Prospective follow-up data of almost 4 years were collected, which allowed for the assessment of longer PE. The effect of training was not demonstrated by a randomised comparison but by comparing baseline (untrained) prognostication abilities with the performance after training in the same group of physicians. The study is a single-centre study—even though performed at a large teaching hospital.
Conclusion
A short and simple training programme was able to increase the prognostic accuracy of residents as well as experienced oncologists significantly. This programme conveyed the basic principles of prognostication, provided freely available epidemiological data and taught the use of simple algorithms.39 This teaching effect was apparent for prognostication of the intermediate future (up to 6 months) but is unlikely to be present beyond this time, especially for time horizons beyond 1 year. However, this does not come as a surprise because: ‘Medicine is a science of uncertainty and an art of probability’ (attributed to Sir William Osler).
Data availability statement
Data are available on reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by Ethics committee of the University of Regensburg and internal review board of the hospital Barmherzige Brüder Regensburg (reference 20-2166-101). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
We thank all members of the medical team of the department of oncology and haematology at the hospital Barmherzige Brüder who participated in this project: Berberich Veronika, Böhm Valentin, Braess Jan, Braun Bernhard, Daum Susanne, Heilmeier Bernhard, Homann Arne, Keyßner Verena, Koch Leonie, Maguire Nadia, Minderjahn Lisa, Moosmann Nicolas, Reiß Kathrin, Schenk Michael, Schlenska-Lange Anke, Seiler Isabella, Stauder Heribert, Stigler Katharina, Stosiek Christoph, Thalmeier Susanne, Wirsching Andreas.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors IK, JB and GT: design and analysis of the study. IK: data collection. IK and JB: writing of the manuscript. MG: provision of survival data. IK and AC: statistical analysis. IK, GT, MG, AC and JB: critical revision of the manuscript. All authors read and approved the final manuscript. IK is the guarantor of the study.
Funding This study was completely funded by the hospital Barmherzige Brüder Regensburg.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.