Information sources

Studies were identified by searching MEDLINE, Embase and Cochrane Library from 1 January 2000 to 28 February 2018. Detailed search strategies for each database are available in the online supplementary data. Highly relevant but excluded articles were recorded, collated and the references analysed for additional studies.

Data management

Search results were exported into Covidence software, an online screening platform endorsed by Cochrane (covidence.org). Title and abstract screening and full-text screening were sequentially performed by a team of three reviewers. Prior to screening, a pilot screening process was conducted for calibration of screening between reviewers. Reviewers were not blinded to study authors, institution, publication journal or year of publication.

Data items

The full list of data items extracted from each included study is recorded in the protocol. These were informed by the CHARMS checklist16 and included: (1) Study design, (2) Characteristics of study participants, (3) Outcomes, (4) Candidate predictors, (5) Sample size, (6) Missing data, (7) Statistical methods, (8) Model performance and evaluation and (9) Usability. Model performance, assessed by discrimination was the principal summary measure.

Bias assessment

To assess the validity of eligible studies, individual studies were assessed for bias using the new Prediction model Risk Of Bias ASsessment Tool (PROBAST).19 The PROBAST tool assesses both risk of bias and applicability of both model development studies and validation studies.20

Patient and public involvement

There was no formalised patient and public involvement in the design or conduct of this review. Preliminary results were presented to a departmental patient and public involvement group, and their comments used to inform the write-up of the review.

Study selection

The search of Cochrane, MEDLINE and Embase yielded 6581 studies after deduplication. Sixteen additional studies were identified by reviewing the references of excluded but relevant studies. A total of 12 studies were eligible for inclusion in the final review.21–32 Two of these had not been summarised in previous reviews.26 27 The PRISMA flow diagram is shown in figure 1, including the reasons for exclusion at full-text screening. Only one reason for exclusion was assigned to each study, when multiple reasons may have been present. Nine of the final 12 included studies were model development studies, three were model validation studies. Two of these external validations were of models already included as model development studies.30 31 One study related to an external validation of the Cancer of the Prostate Risk Assessment (CAPRA) score against mortality.32 The original CAPRA model development study however did not meet the eligibility criteria as it was developed against recurrence rather than long-term survival.11

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

PRISMA flow diagram depicting the flow of information through the different phases of the review. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Study characteristics

Characteristics of the included studies’ participants and settings are summarised in table 1. The model development studies used data from over 231 888 men. However, two studies used analytical cohorts from the same registry (Surveillance, Epidemiology and End Results (SEER)).23 29 Two studies used data from single US centres, two used data from groups of four hospitals, and five were from regional or national multicentre registries. Eligibility criteria of patients into studies varied significantly, with some models using very specific or selected sub-cohorts only. For example Nguyen et al included only consenting men, undergoing radiotherapy (RT) or radical prostatectomy (RP), with at least one intermediate or high-risk feature but ≤T3b disease.22 The treatment cohorts included, and whether treatment effect was a parameter in the final model was highly variable. Only six of the nine models included any men who had been managed conservatively (including ‘watchful waiting’ or ‘conservative management’); none of the models described or defined a specific ‘active surveillance’ cohort.21 23 25 27–29 Only three of the nine models included treatment as a predictor variable - none of which were externally validated.24 25 28 Median follow-up was relatively short within all model development cohorts, with the longest reported follow-up being 7.6 years26.

Results of individual studies

The final results of individual studies are summarised in table 2. The primary outcome was cancer-specific mortality only in one study,27 and was overall-survival only in another study.25 The remainder reported measures of both. The study designs, model-types and performance metrics varied markedly. We therefore focused on describing the studies, results, applicability and availability. Significant heterogeneity in the question being asked and statistical methods of validation between studies meant that attempts to meta-analyse data would not be appropriate. Modelling techniques varied, with the majority of studies using a proportional hazards model (Cox or Fine and Grey) although none reported assessing whether the proportional hazards assumption was valid. Included studies did not report any flexible parametric approaches to deal with continuous variables, and the majority used group categorisations of these variables. Reporting of model accuracy was inconsistent. Considering area under the curve and c-indices synonymously, six of the nine studies reported some measure of discrimination with values ranging from 0.63 to 0.90 for PCa survival outcomes, although this higher figure was derived within a small elderly sub-cohort23 28 and 0.58 to 0.73 for overall outcomes.23 25 Only four of the nine studies reported assessing calibration in some capacity.23 25–27 Relative performance within particular sub-populations were not generally reported.

Validation

Seven of the nine model development studies reported internal validation. Two reported using bootstrapping, and one used a separate 40% random sample of the original dataset for internal validation.21 26 27 An additional three external validation papers were included (table 3). Of note, each of these included the author of the original model within their author list, suggesting these are not completely independent validations. These three models each used large numbers of subjects over comparable timeframes to their model development study. Discrimination of the Cambridge Prognostic Groups, SEER Cancer Survival Calculator and CAPRA scores were comparable for prostate cancer specific mortality (PCSM) at 0.81, 0.81 and 0.80, respectively, over 5 years30-32 (table 4). Discrimination was poorer for overall mortality at 0.71 in the latter study. These external validation papers performed quite well on individual bias assessment (table 5). In the CAPRA validation paper estimates were reported to be ‘adjusted’ for age and treatment type, such that it is unclear whether the reported accuracy reflects that of the usable model.32

Risk of bias

Risk of bias within studies is summarised in both table 5 and figure 2. Frequent concerns were observed with respect to participant selection and inclusion, particularly with respect to reporting or allowing for missing data. The outcome of death was well-defined and unambiguous in the majority of studies. Every included study had at least one parameter for which there was high concern of bias – leading to a high overall judgement of bias in the PROBAST tool. Concerns about applicability to the review question were present in more than half of the studies.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Bias assessment summary (across both the model development and model validation studies).

Usability

All of the included studies reported models that were nomograms, look-up tables or grouping stratifications. Seven of the nine studies were clinically usable through the publication itself. One was also available in a dedicated website.31 The SEER Cancer Survival Calculator was never launched online, and the publication itself does not provide sufficient detail to use the model.23 The model by Margel et al was available but not usable as it included year of entry as a predictor variable, making the model usable only on retrospective series rather than in future individual cases. Indeed, this model was developed to answer the question of whether pathological information adds to a prognostic model, rather than being intended for use at diagnosis.21

Principal findings

Treatment decisions at the point of diagnosis of non-metastatic PCa should be informed by the likely prognosis of the disease. Despite finding a number of published prognostic models, there remains a lack of well-validated, unbiased, generalisable models for use at diagnosis. In particular, there was a lack of external validation and dearth of models that compare outcomes between conservative management and radical treatment.

Prior evidence

A number of previous reviews have assessed ‘prediction’ or ‘risk’ models in their broadest sense.33–35 Shariat et al have previously published a thorough catalogue of available predictive models in PCa – predicting everything from detecting PCa in the initial biopsy setting to survival. Other reviews have focused exclusively on outcomes following radical prostatectomy or radiotherapy.36 37 Lughezzani et al in 2010 for example summarised models predicting Gleason score upgrading, pathological stage, life expectancy, perioperative mortality, postoperative biochemical recurrence or functional outcomes in addition to PCSM after prostatectomy.36 The paucity of models using mortality as an outcome was particularly noted and the study concluded that no tools were capable of quantifying the benefit of RP relative to other treatment modalities.36 In 2009, a separate review by Shariat et al summarised available models into eight groups. One of these groups was of models predicting survival. This group included only four models, all of which related to advanced or metastatic disease requiring hormonal therapy, or androgen-insensitive disease.33 Green et al explored more historical literature from 1966 until 2012.35 They included four studies which looked at life expectancy in men with localised PCa, two of which were included in our study.25 28 The other two were models from Albertsen et al in 1996 which was prior to our study dates, and from Walz et al in 2007 which focused exclusively on non-cancer mortality and therefore did not meet our eligibility criteria.38 39 A recent review into decision-making tools again took an overview of tools available for use at different points in the patient pathway.40 Here, the only mentioned models that predicted survival were all post-prostatectomy models.40 In 2015 Kent and Vickers reported on ‘gross deficiencies’ in current tools for prediction of non-PCa death and concluded that they were unable to identify a suitable life expectancy tool.41 These previous reviews all suggest there is a lack of focus on and availability of good quality survival models.

Interpretation of findings

Although a number of previous reviews have been published, ours represents the most systematic and contemporary work focused towards the decision dilemma that patients and clinicians face, as it includes only models that are available at the pretreatment stage and that are not treatment-specific. This review demonstrates that only a small number of models for this setting have been published using long-term survival as an endpoint, and only three have been externally validated. However, the number of events reported in these three validation studies were well in excess of the 100 suggested as the minimum number needed for adequate validation.42 Within the external validations, model discrimination of up to 0.81 was reported for disease related mortality.30 31 The included studies highlight the potential for using large datasets to develop prognostic models. These have the advantage of providing data from ‘real-world’ settings, outside of the clinical trial context or data exclusively from specialised centres. Our included studies commonly used elements of good study design that would be in keeping with the AJCC acceptance criteria for risk models.13 For example, criteria that were seen in all studies, were that the prognostic time-zero was well-defined, and that model developments have been published in well regarded peer-reviewed journals. However, other criteria were lacking such as reporting measures of discrimination, assessing calibration and thorough reporting of missing data in validation work.

The most robust tools we found are the three that have been externally validated – namely the Cambridge Prognostic Groups, The SEER Cancer Survival Calculator and the UCSF CAPRA score. Of note, the Cambridge Prognostic Group stratification criteria has not previously been summarised in prior reviews, as both its development and validation have been published in the last few years.27 31 However, the SEER calculator has never been released for public use. Each of the other two models have significant shortcomings by disregarding treatment effect, focusing only on disease-specific mortality and ignoring comorbidity such that their value at the point of diagnosis is diminished. Importantly, each of the three external validation studies also had potential flaws in their design, using predominantly complete-case analyses and non-independent authors.30–32

Many of the included studies used historical cohorts. This is a necessity when using long-term survival as an outcome, but, raises issues of generalisability to men diagnosed in the contemporary setting. The uptake of prebiopsy multiparametric MRI and targeted biopsies, may for instance impact on the type of PCa detected and affect generalisability of previous models to current practice.43 Another issue was of small cohorts and low numbers of events in some cases, often as a result of relatively short follow-up. The value of 5 year outcomes themselves is questionable given randomised control trial data on survival in non-metastatic PCa would suggest cancer-survival over this timeframe is incredibly high.44 An important feature of any prognostic model should be its usability and applicability to a man diagnosed today. However, with unquantified missing data this applicability becomes less clear. Dealing with PCSM in isolation may also be problematic, not least because of the importance of competing risks of death in a disease that affects older men.45 In this review we see that all but two included models were derived from North American data, such that generalisability to European or UK men could be questioned, with differing approaches to PSA-testing and different healthcare contexts. This is particularly relevant with regards to screening, whereby historical American cohorts are likely to have been detected through PSA screening, such that issues of lead time bias and detection of cancers that would not be considered clinically-significant may affect model generalisability to modern practice.

Access for clinicians and patients to know about and use models should be easy with modern web-based software. However, we found models were often only available in paper nomograms published with the article. Rather than online resources increasing the availability of models, there is the suggestion of the opposite occurring, with sites such as www.nomogram.org and www.clinicriskcalculators.org which were previously cited in reviews no longer being available online.36 Indeed, the SEER Cancer Survival Calculator, one of the most promising models we explored, which was also externally validated, was never made available online or for clinical use.23 30 The UCSF CAPRA score (www.urology.ucsf.edu) and Cambridge Prognostic Groups (www.cambridgeprognosticgroup.com) on the other hand are freely available online. Online publication does not in itself difficulty in accessing models, or a reluctance to fully share coefficients may partly explain the lack of external validations by researchers outside of the original models’ authorship group. Another hurdle to acceptance will be the ‘face validity’ of a model. For example, in any PCa prognostic model clinicians would likely expect grade, PSA and stage to be incorporated as a minimum set of variables; each of which has been shown to be independently prognostic.46–48 However, two of the models based on the largest dataset failed to include PSA which is inadequately recorded in the SEER database.23 30 A number of the other models used 3-strata grade classifications, rather than the full Gleason grade system, or the contemporary grade group system, which would again seem inadequate for a modern PCa model.49

A key step in the development of any prognostic model, following validation, should be a clinical impact study to quantify whether the model’s use improves decision-making and patient outcomes within a comparative design.50 Reviewing clinical impact studies was beyond the scope of this formalised review, but no such studies were found on simple literature review. These studies not only assess whether use of the model is an improvement on standard care, but also enable the study of factors that may affect implementation into care, such as the acceptability and ease of use to clinicians or patients, which can be difficult to assess in a review such as this.50 Impact studies can also help to bridge the gap between clinical validity and clinical utility, as utility of a model is not proportional to its prognostic capabilities. This has recently been explored further in a review on the UCSF CAPRA score, which confirmed its prognostic capacities, but was unable to demonstrate clinical usefulness – particularly when deliberating between different treatment strategies.51

Strengths and limitations at review level

While this review has particular strengths in its broad coverage of the literature and search strategy, we recognise potential limitations. Although we have assessed bias within individual studies we recognise that risk of bias will also exist across studies, driven particularly by publication bias and selective reporting within studies which we were unable to assess. Other limitations to our review may relate to our timeframe of inclusion of studies only from 2000 onwards. Models developed prior to this time may have undergone more thorough testing or validation and clinical impact assessment. However, our rationale for focusing on this contemporary time period was to investigate models appropriate to modern management; with significant changes having taken place in patient management and diagnostic practice since that time.43 49

We recognise that exciting developments are also underway to propose genomical or biomarker-based prognostic indicators.52 Many of these are currently reported as single parameter studies, rather than being incorporated into existing models. As such these would not meet the eligibility criteria for this review. As others have suggested, any incremental value of these models should be assessed against ‘a gold-standard multivariable clinical prognostic model’.53