Article Text
PDF
Abstract
Objectives As part of the PIONEER Consortium objectives, we have explored which diagnostic and prognostic factors (DPFs) are available in relation to our previously defined clinician and patient-reported outcomes for prostate cancer (PCa).
Design We performed a systematic review to identify validated and non-validated studies.
Data sources MEDLINE, Embase and the Cochrane Library were searched on 21 January 2020.
Eligibility criteria Only quantitative studies were included. Single studies with fewer than 50 participants, published before 2014 and looking at outcomes which are not prioritised in the PIONEER core outcome set were excluded.
Data extraction and synthesis After initial screening, we extracted data following the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of prognostic factor studies (CHARMS-PF) criteria and discussed the identified factors with a multidisciplinary expert group. The quality of the included papers was scored for applicability and risk of bias using validated tools such as PROBAST, Quality in Prognostic Studies and Quality Assessment of Diagnostic Accuracy Studies 2.
Results The search identified 6604 studies, from which 489 DPFs were included. Sixty-four of those were internally or externally validated. However, only three studies on diagnostic and seven studies on prognostic factors had a low risk of bias and a low risk concerning applicability.
Conclusion Most of the DPFs identified require additional evaluation and validation in properly designed studies before they can be recommended for use in clinical practice. The PIONEER online search tool for DPFs for PCa will enable researchers to understand the quality of the current research and help them design future studies.
Ethics and dissemination There are no ethical implications.
- prostate disease
- urological tumours
- epidemiology
Data availability statement
No data are available. not applicable.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A multidisciplinary team including patients, urologists, oncologists, radiation oncologists, methodological experts and pathologists were involved throughout the study.
The search was restricted from 2014 onwards, to maintain a pragmatic approach.
The main strength of this study is the extensive and comprehensive search and screening of the studies included.
Introduction
Prostate cancer (PCa) accounts for 15% of cancers diagnosed1 and is the second most common cancer in males worldwide.2 PCa is clinically and molecularly heterogeneous and is usually suspected based on the clinical findings of digital rectal examination and/or prostate-specific antigen (PSA) levels.1 However, which diagnostic or prognostic factors (DPFs) can be used to select patients for specific therapeutic options remains largely unclear.3 Specific biomarkers in urine or in blood are available on top of traditional PSA testing, such as PCA3, TMPRSS2-ERG fusion or kallikreins as incorporated in the Phi or 4Kscore test together with other parameters including family history.4–7 However, the European Association of Urology (EAU) guidelines (2021) currently do not provide general recommendations to implement these biomarkers into routine screening programmes due to limited data. As part of the American Society of Clinical Oncology (ASCO) guidelines, Eggener et al recommended commercially available biomarkers, which have been shown to provide prognostic significance and additional information beyond standard clinical models in patient selection in the localised context: Oncotype Dx Prostate, Prolaris, Decipher, and ProMark.8 However, no guidelines have recommended DPFs for other stages of PCa. The expert panel at the Advanced Prostate Cancer Consensus Conference (APCCC) consensus meeting of advanced PCa in Basel 2019, recommended AR-V7 for mCRPC as potentially useful, which ultimately led to the inclusion of AR-V7 testing in the NCCN guidelines.9
The PIONEER Consortium is an international collaboration coordinated by the EAU, which aims to establish the best evidence-based management and clinical practice of PCa across all disease stages using the power of big data analytics towards a more outcome-driven, value-based and patient-centric healthcare system.10 A key objective is to address one of the major challenges within the context of diagnostic or prognostic biomarkers/factors: the inability to incorporate DPFs into the management of PCa in terms of screening, diagnosis and treatment. It is therefore important to summarise and evaluate the evidence. Biomarkers can be classified into different types: diagnostic, prognostic, predictive and therapeutic—in this study we focus on the first two.11 A diagnostic biomarker or factor is useful when cancer is suspected and allows the early detection based on symptoms or tests.11 The overall aim of a diagnostic biomarker is to distinguish people with the diseases from people without the disease. A prognostic biomarker or factor is a clinical or biological characteristic which provides information on the likely course of the disease, that is, biochemical progression or disease recurrence.11 It enables clinicians to decide on the most suitable treatment depending on the likely course of the disease. In the sections below, we have used the terms biomarkers and factors interchangeably. Multiple DPFs can be measured in tissue, blood or urine. These come with different advantages and disadvantages and only a limited number of factors are currently available for PCa in standard clinical care.
We aimed to systematically review the evidence from 2014 onward to assess which DPFs are available in relation to previously defined outcomes for PCa.
Methods
The systematic review (SR) followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.12 A detailed protocol of the overall project was published elsewhere13 (please see the protocol attached as methods online supplemental appendix). Briefly, we followed the following four steps (figure 1):
Supplemental material
Comprehensive systematic literature review of DPFs for all stages of PCa (localised, locally advanced, metastatic, and non-metastatic castration resistant) from 2014 onwards. DPFs developed before 2014 were not included, due to the significant changes influencing the staging of PCa (i.e., Consensus Conference on Gleason Grading of Prostatic Carcinoma [60]) that have taken place in diagnostic and prognostic practice and patient management since then.
Assessment and identification of final list of DPFs by a multidisciplinary expert panel.
Evaluation of quality of studies published using risk of bias (RoB) tools: Prediction model RoB Assessment Tool (PROBAST) if applicable; or Quality in Prognostic Studies (QUIPS) tool for prognostic and the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool for diagnostic factors.
Due to the heterogeneity of the studies identified no further formal quantitative assessments in the form of a meta-analyses could be performed. Hence, the findings of stages 1–3 have been reported here as the results of a SR.
Overview of four stage process. CHARMS-PF, Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of prognostic factor studies; DPFs, diagnostic and prognostic factors; PROBAST, Prediction model Risk Of Bias Assessment Tool; QUADAS-2, Quality Assessment of Diagnostic Accuracy Studies 2; QUIPS, Quality in Prognostic Studies; SR, systematic review.
Stage 1: comprehensive literature review
We developed the search criteria for the first search with an information scientist who specialises in SR for urology. MEDLINE, Embase and the Cochrane Library were searched on 21 January 2020. The second search was developed following a consultation with an independent information scientist group who excluded row 12, 14 and 16 of (see online supplemental table 1). We screened the EAU Guidelines reference list for PCa in our third search (see figure 2).
Supplemental material
PRISMA flow diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; COS, Core outcome set.
Stage 2: multidisciplinary expert meeting
On the 20 March 2020, we invited a group of multidisciplinary participants to discuss the identified articles on DPFs (see online supplemental table 2). The participants were presented the search criteria and the extracted data. Data extraction followed the CHARMS-PF checklist and we added author and year of publication.
Supplemental material
Stage 3: evaluation of quality of studies published using the RoB tools
Prior to the evaluation of the quality of studies, an initial pilot screening to prepare the raters for the use of PROBAST, QUADAS-2 and QUIPS was performed. This aimed to reach consensus on how to judge the domains of the assessments using the three RoB tools. Two urologists (FB and SS) and two epidemiologists (AH and KB) were involved in the pilot assessments. The group discussed any discrepancies. Articles which presented the development and validation either internal validation or external validation (i.e., the same data was used for both development and internal validation, such as bootstrapping or cross-validation; different populations were used for development and validation), of a diagnostic or prognostic model were assessed with PROBAST. Papers assessing single biomarkers or with/without validation were assessed with QUIPs for prognostic or QUADAS-2 for diagnostic biomarkers.
Evaluation of quality of studies published using QUADAS-2
The RoB of diagnostic factors without validation or single validated factors was evaluated using QUADAS-2. We assessed the following four domains: patient selection, index test, reference standards and flow and timing. The first three domains are assessed looking at applicability and all four domains were assessed in terms of RoB.14 We created a summative score after the diagnostic studies were assessed by two reviewers and in case of disagreement a third reviewer assessed the study.
Evaluation of quality of studies published using PROBAST (diagnostic)
The RoB of internal or external validated diagnostic models was assessed using the PROBAST RoB tool. PROBAST includes four domains assessing the RoB (i.e., participants, predictors, outcome, and analysis) and four domains assessing applicability (i.e., participants, predictors, and outcome) (see online supplemental table 3 for scoring information).
Supplemental material
Evaluation of quality of studies published using QUIPS
To assess the articles which are single factors or were not internally or externally validated, we used the QUIPS rating procedure (see online supplemental table 4 for scoring information). To standardise the approach across raters, we used the QUIPS electronic spreadsheet (excel) from Hayden et al.15 There are no rules available for QUIPS on how to score the overall RoB of a paper. Due to the large number of papers and the need for synthesis, we followed the suggestions from Grooten et al., and categorised on the following criteria: (1) Paper was classified as low RoB if all domains were classified as having low RoB, or up to one moderate RoB; (2) Paper was classified as high RoB if one or more domains were classified as having high RoB, or ≥3 moderate RoB; (3) Paper was classified as having moderate RoB if all papers in between 1 or 2 (see online supplemental table 1). This assessment was based on the risk scores of individual assessments within the group. If the overall assessment was not possible due to differences in the individual category, a third assessor reviewed the assessments and the results were discussed.
Supplemental material
Evaluation of quality of studies published using PROBAST (prognostic)
The RoB of prognostic validated models were assessed using PROBAST. As highlighted above, PROBAST includes four domains assessing the RoB (i.e., participants, predictors, outcome and analysis) and the domains assessing applicability (i.e., participants, predictors and outcome).
Results
Stage 1: comprehensive literature review
Stage 1 identified 6604 citations and contained three independent searches. After removing duplicates, we screened 4215 abstracts, from which 489 met the inclusion criteria.
Stage 2: multidisciplinary expert meeting
The group discussed the results and additional literature on DPFs was suggested to help the classification of the DPFs, such as the ASCO Guideline on Molecular Biomarkers in Localised Prostate Cancer.16
Stage 3: evaluation of quality of studies published using the RoB tools
The 489 articles were equally divided between six groups. The six groups received the guidance documents which were identified during the pilot phase.14 15 17–19 In addition, MvH and KB discussed questions with each individual group.
Evaluation of quality of studies published using QUADAS-2
The RoB of the 41 included studies was low for 10 studies, high for 23 studies and unclear for eight. RoB concerning applicability was low for 10 studies, high for 21 studies and unclear for 10 studies (see table 1). Table 2 shows the studies with an overall low RoB across both categories. Two studies were identified to have an overall low RoB.20 21
Overall judgement of RoB
Non-validated DPFs with overall low RoB: QUADAS-2
Evaluation of quality of studies published using PROBAST (diagnostic)
We identified 20 papers to be assessed with PROBAST. The RoB of three papers was low, high for 14 and was unclear for three. The applicability of eight papers was high and was unclear for two (see table 1). Online supplemental table 1 shows the criteria on how to judge the RoB. One study had an overall low RoB across both domains. All categories except ‘predictors’ was scored to have a low RoB. There was little information available for the category predictors and therefore it was scored as ‘unclear’ (see table 3).
DPFs assessed with PROBAST
Evaluation of quality of studies published using QUIPS
The 12 assessors independently inserted the relevant information and assessed each domain such as participation, attrition, prognostic factor confounding and statistical analysis and reporting.
A total of 387 prognostic factors were assessed using QUIPs. A total of 307 papers were classified as high RoB. Forty-nine papers were classified as having a moderate RoB and 28 papers were scored as low RoB (see table 1). Out of the 28 papers with a low RoB, the most common moderate bias was linked to attrition (12 papers), followed by confounding (4 papers), participation (3 papers), outcome (1 paper), statistical analysis (1 paper) (see table 4).
Characteristics of DPFs with overall low RoB
Evaluation of quality of studies published using PROBAST (prognostic)
The assessors identified 44 papers to be assessed with PROBAST, of those three scored a low RoB, 27 a high RoB and 13 were assessed as unclear (see table 1). In terms of applicability, 15 papers scored low, 20 high and eight unclear. Two papers were scored to have an overall low RoB22 23 (see table 3).
Characteristics of studies identified with low RoB
Details of the identified validated DPF models with an adequate quality are presented in table 5. We identified 32 studies with an overall low RoB (assessed with PROBAST, QUIPS, QUADAS-2). Out of these 32 studies, we identified one validated diagnostic model (assessed with PROBAST),24 two validated prognostic models (assessed with PROBAST),22 23 two non-validated diagnostic single factors (assessed with QUADAS-2)20 21 and 26 prognostic factors (assessed with QUIPS)20–50 which have not been validated and two single prognostic factors which have been validated (assessed with QUIPS).34 50 Prognostic factors assessed with QUIPS were identified with a low RoB for the localised PCa population. Sixty-seven per cent of the low RoB DPFs were intended to be measured after the treatment was performed. In addition, the most commonly measured outcome was biochemical recurrence followed by overall survival. However, it is important to take into consideration that even from the studies assessed with a low RoB, only 2 out of the 32 were of a non-observational study design.
DPFs with low risk of bias assessed with QUIPS
As highlighted above, we identified three validated DPFs which were scored to have a low RoB and low risk concerning applicability. First, we identified the ‘Unified Prostate Cancer Risk Prediction Model Combining the Stockholm 3 Test and MRI’, a risk prediction model which combines clinical variables, genetic and protein biomarkers. Five hundred and thirty-two men were involved across three centres.24 Second, the DREAM challenge developed a set of five standardised raw event-level tables, using laboratory values, patients’ demographic information, medical history, lesion sites, previous treatments and vital signs of patients with mCRPC. These variables where combined by using data from four clinical trials.22 Third, Joniau et al., developed ‘Pretreatment Tables’ to predict the pathologic stage of locally advanced PCa after RP based on pretreatment PSA level and biopsy Gleason score.23
We identified two single factors which were validated and had low RoB. First, Lara et al., assessed and validated the serum biomarkers of bone metabolism (N-telopeptide and pyridinoline) and formation (C-terminal collagen propeptide and bone alkaline phosphatase)) in 778 CRPC patients as part of the randomised phase III SWOG trial (S0421) of docetaxel/prednisone with or without atrasentan.34 Second, Berg et al., showed that ERG expression can be used to estimate the risk of progression during AS including 265 patients at diagnosis and progression during AS.50
Discussion
Despite the large number of studies on DPFs which are published every year, there is a paucity of DPFs that are suitable to be incorporated into clinical practice. The majority of DPFs have not yet been validated and are identified in poor quality studies. Our analysis found that most identified studies had a high to moderate RoB due to poor design standards, conduct, reporting and/or analysis that is, generalisability and size of the population, poor model development (no testing or missing important confounders) or only correlation studies, missing data was rarely reported. However, we did identify a small number of validated DPFs with low RoB. We identified three validated models which combine: first, clinical variables, genetic and protein biomarkers, and improved clinical outcome performance of PCa diagnostics (The Unified Prostate Cancer Risk Prediction Model)24; second, laboratory values, patients’ demographic information, medical history, lesion sites, previous treatments and vital signs of patients with metastatic castration-resistant PCa (DREAM challenge)22; and third, pretreatment PSA level and biopsy Gleason score to predict the pathological stage of locally advanced PCa (‘Pretreatment Tables’).23
Two single factors have been validated: the serum biomarkers of bone metabolism in CRPC patients34 and the ERG expression, which can be used to estimate the risk of progression during AS,50 which has already been highlighted in the clinical guidelines.1
Aladawani et al., assessed prediction models for PCa to be used in primary care settings in their SR and identified five models which met their inclusion criteria. From these identified models only one model was externally validated and only one (the Lazzari model 251 had the potential to be implemented in primary care. Lazzari et al., had the lowest RoB (based on PROBAST); however, it must be externally validated before it can be implemented. Hence, Aladawani et al., also concluded that the existing models have limitations concerning study design and reporting performance.52
Tian et al., conducted a review on biomarkers for CRPC patients, however, their quality assessment was focused on study design (RCT vs. observational study), whereas we focused on biomarker specific tools.53 While Tian et al., and our review identified similar factors and quality scores, there were slight discrepancies between the overall RoB assessments. Tian et al., used an overall quality assessment scale from 1 to 6 instead of low, medium and high. In their assessment the validated prognostic study by Lara et al.,34 and the non-validated prognostic factor by Pei et al.,41 were scored on the quality scale as 4 (medium quality). We assessed Lara et al.,34 to have a low RoB with a moderate risk of confounding and Pei et al.,41 with a moderate RoB concerning the prognostic factor itself. This might explain the discrepancies between the two quality assessments. The reports by Alvim et al., Qu et al., were assessed to have the highest quality by Tian et al.,53 similar to our review. This illustrates that different quality assessment tools emphasise different criteria, which may result in small discrepancies. However, the overall conclusion for prognostic single factors was similar in our review and to the work of Tian et al.53
Similar issues have been identified for other urological cancers. For example, in kidney cancer, a large body of research was identified by Harrison et al., with very few validated studies and lots of heterogeneity.54 Schmitz-Dräger et al., published an International Consultation of Urologic Disease/WHO Consensus manuscript where they identified that in bladder cancer one of the main limitations for the lack of incorporation of modern bladder cancer tests into clinical practice decision making is linked to the scarcity of ‘good clinical practice guidelines’ for the evaluation of diagnostic markers.
There is a need for improved guidance on development and validation of diagnostic markers.55 To meet that need, we are developing the PIONEER DPF search tool, which will help researchers and clinicians to get a better understanding of the DPFs for PCa. The tool will not only summarise all relevant studies, but also provide information on the use and results of different RoB assessment tools, which will enable an understanding of the quality of published studies.
Future research should, therefore, focus on addressing the identified shortcomings such as heterogeneity, validation and poor RoB by designing more robust studies which consistently include RoB assessments such as PROBAST, QUIPS or QUADAS-2.
With the growing number of various therapeutic options, diagnosis and management of PCa requires an individualised approach to patient care. There is an unmet need for DPFs to guide decisions for optimal treatment and to predict which patients will benefit the most, from a particular management strategy. DPFs could potentially enhance the quality of patient counselling, but currently most need additional evaluation and validation in properly designed studies. Our SR highlights the need for well-designed Real-World Evidence studies, while the PIONEER online search tool can inform the design of new research studies, through providing a rigorous evaluation of the methodological quality of the studies.
The main strength of this study are the extensive and comprehensive search and screening of the studies included. In addition, we are developing an online search tool which showcases the identified and assessed studies. It provides an overview of the available DPFs and enables interested stakeholders to search for DPFs. To our knowledge, this is the first study which has been performed with this extensive amount of literature.
Patient and public involvement
This project has been overseen by a multistakeholder group part of the PIONEER Consortium. PIONEER brings together 35 key stakeholders from academic institutions, patient advocacy groups, European organisations, experts in legal data management, clinicians and pharmaceutical companies, as well as regulatory agencies, economics and ethics, and information and technology specialists. Patients and their family members are therefore involved and actively participate as an integral part of all research conducted by the PIONEER Consortium.
Limitations
Even though this review included three searches and assessments by a multidisciplinary group of fourteen researchers, we recognise potential limitations. Studies were only included from 2014 onwards and DPFs developed before 2014 were not included. However, significant changes which influence the staging of PCa (i.e., Consensus Conference on Gleason Grading of Prostatic Carcinoma56 have taken place in diagnostic and prognostic practice and patient management. This changed the staging of the patient population and therefore has an impact on DPFs.
In addition, there is a potential of subjectivity in the evaluation of the studies. Even though the studies have been assessed in duplicate, there might be variation across groups. However, given the overall moderate to high RoB, this does not influence the overall recommendation of the project.
Conclusion
At present DPFs that are capable of significantly improving diagnosis and prognosis in PCa are an unmet need as most of the DPFs identified require additional evaluation and validation in properly designed studies before they can be recommended for use in clinical practice. Well-designed real world evidence (RWE) studies can help to increase quality. Our SR aims to inform clinicians and patients about this rapidly evolving field, while the PIONEER online search tool for DPFs for PCa will enable researchers to perform future research, and to understand the quality of the current available studies.
Data availability statement
No data are available. not applicable.
Ethics statements
Patient consent for publication
Ethics approval
This study does not involve human participants.
Acknowledgments
PIONEER Consortium
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
KB, LM and ML are joint first authors.
Twitter @beyer_katharina, @prostatePioneer, @Ric_Campi, @endourologist, @drimranomar, @alexhabs, @SajMacLennan
Contributors KB, LM, ML, AH, FB, SS, MM, MIO, SM, MJR, BF, EV, ZD, AA, JZ, SJM, LC, JN; ABr, ABj and MVH conceptualised designed the review. Abstracts and full texts were reviewed and data extracted by KB, LM, ML, AH, FB, SS, MM, RH, AR, RC, IG, KS, TvdB and SA. Authors resolved disagreement by discussion where necessary. The risk of bias was assessed by KB, LM, ML, AH, FB, SS, MM, RH, AR, RC, IG, KS and SD. The manuscript was drafted by KB, LM, ML, ABr, MVH and reviewed by KB, LM, ML, AH, FB, SS, MM, RH, AR, RC, IG, KS, SD, TvdB, MG, GG, MIO, SM, MJR, BF, EV, ZD, AA, JZ, SJM, LC, JN, ABr, ABj and MVH. The whole project was supervised and guided by JZ, SJM, LC, JN, ABr, ABj and MVH. PIONEER Consotium acts as a gaurantor.
Funding PIONEER is funded through the IMI2 Joint Undertaking and is listed under Grant Agreement No. 777492 and is part of the Big Data for Better Outcomes Programme (BD4BO). IMI2 receives support from the European Union’s Horizon 2020 research and innovation programme and the European Federation of Pharmaceutical Industries and Associations (EFPIA). The views communicated within are those of PIONEER. Neither the IMI nor the European Union, EFPIA, or any Associated Partners are responsible for any use that may be made of the information contained herein.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.