Article Text

Download PDFPDF

Evaluating methodological quality of Prognostic models Including Patient-reported HeAlth outcomes iN oncologY (EPIPHANY): a systematic review protocol
  1. Nina Deliu1,
  2. Francesco Cottone1,
  3. Gary S Collins2,
  4. Amélie Anota3,
  5. Fabio Efficace1
  1. 1 Data Center and Health Outcomes Research Unit, Italian Group for Adult Hematologic Diseases (GIMEMA), Rome, Italy
  2. 2 Centre for Statistics in Medicine, University of Oxford, Oxford, UK
  3. 3 Methodology and Quality of Life in Oncology Unit (INSERM UMR 1098), University Hospital of Besançon, Besançon, France
  1. Correspondence to Dr Fabio Efficace; f.efficace{at}


Introduction While there is mounting evidence of the independent prognostic value of patient-reported outcomes (PROs) for overall survival (OS) in patients with cancer, it is known that the conduct of these studies may hold a number of methodological challenges. The aim of this systematic review is to evaluate the quality of published studies in this research area, in order to identify methodological and statistical issues deserving special attention and to also possibly provide evidence-based recommendations.

Methods and analysis An electronic search strategy will be performed in PubMed to identify studies developing or validating a prognostic model which includes PROs as predictors. Two reviewers will independently be involved in data collection using a predefined and standardised data extraction form including information related to study characteristics, PROs measures used and multivariable prognostic models. Studies selection will be reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, with data extraction form using fields from the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist for multivariable models. Methodological quality assessment will also be performed and will be based on prespecified domains of the CHARMS checklist. As a substantial heterogeneity of included studies is expected, a narrative evidence synthesis will also be provided.

Ethics and dissemination Given that this systematic review will use only published data, ethical permissions will not be required. Findings from this review will be published in peer-reviewed scientific journals and presented at major international conferences. We anticipate that this review will contribute to identify key areas of improvement for conducting and reporting prognostic factor analyses with PROs in oncology and will lay the groundwork for developing future evidence-based recommendations in this area of research.

Prospero registration number CRD42018099160.

  • patient-reported outcomes
  • prognosis
  • cancer
  • survival

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

View Full Text

Statistics from

Strengths and limitations of this study

  • This systematic review will critically evaluate the methodological quality of recent studies in oncology in order to examine the robustness of patient-reported outcomes (PROs) as independent prognostic factors for overall survival (OS). Compared with the existing literature, it will extensively consider and evaluate recently published studies in all cancer sites and all types and measures of PROs.

  • The main strength of our review is the amount of relevant information, related to both clinical outcomes and methodological issues, collected through the designed data-extraction form.

  • Another key strength of our review is that it will contribute to identify key areas of potential improvement for conducting methodologically sound prognostic modelling analyses, and would contribute to healthcare delivery; for example, by improving management of patients in clinical practice and by informing a more accurate stratification of patients in clinical trials.

  • The main limitation of this systematic review might be the heterogeneity of selected studies in terms of study design and study population, therefore limiting our ability to provide strong recommendations for a specific cancer population.

  • An additional limitation of this study is that we will only search one database.


Patient-reported outcomes (PROs) play a key role in cancer research in addition to traditional clinical outcomes, such as overall survival (OS) or tumour response.1 2

A PRO is an umbrella term covering a measurement of any aspect of a patient’s health status directly reported from the patient, and it is now included by the US, Food and Drug Administration (FDA) among the four types of Clinical Outcomes Assessment (COA) measures that can be used to determine whether drugs provide a treatment benefit.3 The value of including PROs as endpoints in comparative studies is that of generating data that can be used to more robustly inform patient care. The increased use of PROs in clinical trials4 also reflects a growing recognition by major international health policy and regulatory authorities, of the importance of PROs in providing additional information about benefits, risks and cost of therapy, thus supporting claims in approved medical product labelling.5–7

While PROs are frequently included in clinical trial settings for treatment outcome comparisons, a more recent line of research has indicated that PROs, including functional aspects and symptomatic dimensions, can also provide additional prognostic information for survival outcomes. Furthermore, in some specific cancer populations, it has been shown that prognostic accuracy of well-established and validated prognostic models, based on laboratory and clinical data, can be improved by adding PROs.8

One of the most comprehensive critical reviews of studies investigating the prognostic value of baseline PROs on patients’ survival was reported by Gotay et al.9 Notably, out of the 39 clinical trials reviewed and published from 1989 to 2006, 36 studies found at least one PRO associated with OS in the multivariable analysis. PROs most frequently associated with OS are global quality of life (QoL) and physical functioning (PF). Further evidence that PROs provide additional prognostic value beyond traditional clinical measures in patients with cancer has been reported in several secondary pooled analyses of EORTC clinical trials.10–13 For example, a large meta-analysis published in 2009 by Quinten et al,11 comprising 30 randomised controlled trials (RCTs), including data for >7000 patients with a variety of cancers, found that including both traditional predictors and PROs in multivariable analyses increased the ability to predict OS by 6% compared with traditional information alone. These findings have been replicated across other cancer populations, further strengthening the scientific grounds of this association.14–23 It is worth highlighting that diverse PRO questionnaires are often used which has frequently hindered clear comparisons across studies.9


The completeness of reporting of studies describing the development or validation of prediction models in medicine, in general, is known to be suboptimal and major efforts have been made in recent years to raise standards and transparency of such studies. The Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Statement has been recently developed to guide authors of reports describing prediction models.24

While there is a substantial body of literature documenting the association between PROs and survival outcomes, there is also an indication that the design, modelling techniques and reporting of many of these studies are frequently plagued by a number of methodological shortcomings.25 The inclusion of PROs into studies of prognosis holds several major challenges. As PROs questionnaires are actually developed to measure the patient’s perspective of disease or treatment burden, they typically include multidomains which often measure aspects of a patient’s health status that might be, more or less, correlated with each other. Indeed, one of the key methodological challenges is the possible harmful multicollinearity among questionnaire subscales entering the analysis, which may negatively affect the  stability of the final prognostic model.26

One of the most rigorous reviews examining the methodological quality of studies evaluating the prognostic value of PROs was published by Mauer et al.25 In this systematic review (which included studies published between 1989 and 2006), the authors found that most analyses were undertaken retrospectively, and on an exploratory basis, often not preplanned in the original study protocol. Also, the selection process for the prognostic factors was rarely described, and only in one study,27 for example, the total number of included predictors relative to the total number of deaths, as recommended by Harrell et al,28 was checked. Correlations among all PRO domains included in the model were rarely investigated, therefore limiting a critical appraisal of the potential impact of multicollinearity in final model selection.26 29 Finally, most studies lacked an assessment of the added prognostic value of individual PROs as compared with traditional clinical factors alone, and none of the publications reviewed by Mauer et al 25 reported an external validation of the multivariable model (ie, an evaluation of model performance on a new dataset of patients).

While the independent prognostic value of PROs has been documented across many cancer disease sites and in several studies, the lack of methodological and analytic rigour makes current evidence far from being conclusive, likely undermining the translation of findings into real-world practice applications.25 Therefore, a critical evaluation of the methodological quality of most recent studies is needed in order to examine the robustness of PROs included in prognostic models for OS.


The aim of this systematic review is to critically investigate and report the quality of studies assessing the prognostic value of PROs for OS to identify methodological areas in need of particular attention and to provide evidence-based recommendations.

The specific aims of this systematic review are:

  • To investigate the methodological quality of cancer prognostic modelling studies incorporating PRO data.

  • To examine whether the methodological quality of these studies has improved over time.

  • To describe the prognostic significance of PROs across different cancer types, stages and treatments.

Methods and analysis

This systematic review protocol adheres to the preferred reporting process and checklist outlined within the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols 2015 (PRISMA-P) guidelines30 (online supplementary file 1). In accordance with these guidelines, this systematic review protocol has been registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 28 June 2018 under registration number CRD42018099160. At the time of PROSPERO submission, formal screening of search results against eligibility criteria was ongoing. Any amendments to the study protocol will be documented contemporaneously on the PROSPERO database site.

Supplementary data

Eligibility criteria

Types of studies

According to the checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS) checklist31 the proposed review will include:

  • Prognostic model development studies without external validation on independent data.

  • Prognostic model development studies with external validation on independent data.

  • External model validation studies.

As a comprehensive review on the prognostic role of PROs for survival has already been published outcomes in 2008,25 this systematic review will be based on the most recent literature, that is, previous 5 years (January 2013 to December 2017). All studies’ designs, thus also secondary data analyses, will be considered for inclusion. This specific time window for the identification of studies published within the previous 5 years was selected in order to provide state of the art evidence in this area and to also possibly compare our recent findings with previously published work, reporting on older studies.25 Only English language articles will be considered. Grey literature such as reports or working papers, conference presentations, opinions or commentaries and literature reviews will be excluded.


We will include studies on patients with a confirmed diagnosis of any cancer type, whatever stage and treatment(s) received. Both male and female adult patients (aged ≥18 years) will be considered.

Types of prognostic models

We will assess multivariable prognostic models and validation studies developed for patients with cancer with the following characteristics:

  • OS as the outcome to be predicted, where OS is defined as the time from a prespecified time point (eg, date of randomisation, date of diagnosis and date of start of treatment) until death from any cause. Models predicting survival outcomes other than OS, for example, disease-free survival, progression-free survival or survival rate, will not be considered for inclusion.

  • At least one baseline PRO as a predictor, defined as a patient-reported indicator of well-being and measured with a multidimensional instrument. A baseline PRO might be a PRO collected at any fixed predefined time point, for example, before the start of intervention, before randomisation or, in general, at study entry. Studies reporting only changes in PRO scores will be excluded, as such measures may reflect rather than predict changes in survival time related to disease status.

Studies examining the association of baseline PROs with OS in a univariate or unadjusted fashion analysis only, without controlling for sociodemographic or clinical parameters, will not be included in this systematic review.

Search strategy

An electronic search strategy will be performed in PubMed to identify studies that match our eligibility criteria. As specific search filters are not available neither for PROs nor for prognostic studies, available published filters will be adapted and combined for a highly sensitive search strategy.32 We will use medical subject heading (MeSH) and free-text words related to PROs and broad statistical strategies to identify prognostic modelling studies incorporating at least one PRO parameter and limited to adult (≥18 years old) cancer populations. Details on the key searching strategy used are reported in the online supplementary file 2 (PubMed key search).

Supplementary data

Recognising potential limitations of electronic search strategies, we will supplement our search to identify potentially relevant studies from other sources, including reference lists of included studies, index-related articles on PubMed and existing relevant reviews.

Study selection and data extraction

Articles identified through database searching will be exported from PubMed into Microsoft Excel and duplicate records will be removed electronically. Record screening will occur in the exported Excel file to ensure that all retrieved references are fully tracked. One reviewer will compare the titles and abstracts yielded by the search against the inclusion criteria and will select articles for a full-text review. A second reviewer will confirm articles identified as eligible for inclusion, and reasons for exclusion will be documented for all excluded articles. The selection process, including search results and the reason for exclusion, will be reported in a flow diagram, as recommended in the PRISMA statement.33

At least two trained reviewers will independently be involved in the data extraction of studies meeting the eligibility criteria, which will be uploaded in a Google Drive folder for complete review. Data collection will be performed electronically in the non-proprietary Research Electronic Data Capture (REDCap, Vanderbilt University, USA; platform,34 which is a secure web application developed for building and managing online surveys and databases. Each reviewer will be provided with a personal password to access the study website and complete a standardised electronic data extraction form (eDEF).

The eDEF will collect information related to study characteristics (eg, study design/source of data, study objectives and study population) and PROs (eg, type of questionnaire used, handling of PROs in multivariate analysis, compliance and their association with OS). It will also include fields relevant to multivariable models, based on CHARMS31 and TRIPOD24 checklists, including modelling method, handling of predictors, predictors selection process, model performance and evaluation.

Items included in the eDEF were piloted in a sample of similar previously published (ie, prior January 2013) articles to assess their clarity and applicability. The full list of items included in the eDEF (see box 1) along with a comprehensive guideline on how to evaluate them is available to reviewers. This guideline is meant to minimise the number of discrepancies arising from the reviewing process. Should disagreements arise regarding extracted data, the reviewers will discuss and revisit the paper to reconcile any differences until consensus will be achieved. The reconciliation process will be facilitated by consulting a third senior reviewer or contacting study authors, if necessary. The final eDEF will then be validated by the project manager and locked for a final analysis.

Box 1

Domains and items of the EPIPHANY data extraction form

  1. Analysis type

    • Type of study analysis.

  2. Study characteristics (model development)

    • First author.

    • Journal title.

    • Year of publication.

    • Cancer type.

    • Cancer stage.

    • Broad treatment type.

    • Was this study a secondary data analysis?

    • Source of data/study design.

    • Was the prognostic model incorporating PRO the primary objective of the study?

    • If no, what was the primary objective?

    • If yes, was sample size determination based on the primary objective (ie, prognostic model development/validation)?

    • Primary outcome predicted by the prognostic model.

    • Secondary outcomes predicted by the prognostic model.

    • Was the prognostic model with PRO identified in the abstract?

    • Any rationale for including PRO in the prognostic model?

    • Please specify when were baseline PROs collected.

    • Type of PRO instruments used.

  3. Multivariable model and predictors (model development)

    • Multivariable model used.

    • Were model assumptions assessed and reported.

    • Was the number of events (deaths) checked in relation to the number of candidate predictors (Events per Variable)?

    • The rationale for the choice of PRO candidate variables used to build the final model.

    • Strategy to build the final model.

    • If stepwise, forward or backward, which stopping criterion was used?

    • Was multicollinearity among baseline PRO variables investigated?

    • If yes, which technique was used to account for baseline PRO multicollinearity?

    • Were any interactions among predictors examined?

    • How were continuous baseline PRO predictors handled?

    • If any continuous baseline PRO predictor, did the authors report its range?

    • For continuous baseline PRO that was categorised/dichotomised, how was this done?

    • Have the authors been clear on the number of categories of baseline PRO predictors?

    • Did the authors evaluate the performances of the prognostic model?

    • Which kind of measures were used to evaluate model performance?

    • Any additional analyses carried out?

  4. Missing data (model development)

    • Were missing covariates mentioned in the paper?

    • The extent of missing baseline PRO questionnaires was reported.

    • If yes:

    • Report the number of returned PRO questionnaires.

    • Report the number of due PRO questionnaires.

    • The extent of missing baseline PRO data was given for each scale separately.

    • Is there any evidence of comparison of baseline sociodemographic and clinical characteristics of patients with and without completed baseline PRO questionnaires?

    • Is there any evidence of comparison of survival between patients with and without completed baseline PRO questionnaires?

  5. Results (model development)

    • Overall number of patients analysed.

    • Age of patients.

    • Follow-up time.

    • Number of events (ie, deaths).

    • The overall number of candidate covariates considered prior to any model building and/or univariate screening.

    • Number of baseline PRO candidate covariates considered prior to any model building and/or univariate screening.

    • If (multi)collinearity was investigated:

    • Was (multi)collinearity among baseline PRO variables detected?

    • If yes, please specify (multi)collinear PRO covariates.

    • If yes, was any PRO excluded from model building process for (multi)collinearity reasons?

    • Total number of candidate variables entering in the multivariable model.

    • Total number of baseline PRO candidate variables entering the multivariable model.

    • Were any known risk factors forced into the model prior to building the final model?

    • Was the prognostic model adjusted for sociodemographic or clinical characteristics?

    • The overall number of predictors included in the final model.

    • Did the final model include any baseline PRO predictor?

    • If Yes, please report the number of baseline PRO predictors included in the final model.

    • Instruments of baseline PRO included in the FINAL model.

    • For each PRO included in the final model, please report the corresponding regression coefficient/HR.

    • For each PRO included in the final model, please report the corresponding SD/CI.

    • Please specify the confidence level of the reported CI.

    • Did the authors reported HRs for the number of points increase different from one unit (eg, for every ten-points increase)?

    • Were p values reported for all baseline PRO candidates entering in the multivariable model?

    • Standard errors or CIs reported for all coefficients or HRs of baseline PRO candidates entering in the multivariable model?

    • Was a prognostic index developed?

    • Please report the value obtained for model performance.

    • Comparison of model performance with and without baseline PROs was done.

    • If yes, did baseline PROs increase predictive accuracy?

  6. Model validation

    • Did the authors also validate the model?

    • If yes, how was this done?

    • Did the authors evaluate the performances of the validation prognostic model?

    • If yes, Which kind of measures were used to evaluate validation model performance?

    • Did the authors discuss or mention concerns of overfitting/optimism?

    • Have any modifications been suggested to the model based on the validation process?

    • Based on the validation findings, did the authors refit or recalibrated the model?

    • If the authors updated the model, briefly describe what did they do.

  7. Model validation (In case of external validation only)

    • Were any of the authors of the validation study an author of the development study?

    • Is the outcome (ie, OS) of the validation study different from that of the development study?

    • Please report OS of the population of the validation study.

    • Did the authors discuss the distribution of baseline patients’ characteristics of the validation study in comparison to those of the development study (either discussed or in tabulation)?

    • If yes, did they report that they are:

    • Were missing baseline PRO data mentioned in the validation study?

    • Were baseline PRO predictors coded the same as in the development study?

    • If no, please report what were the discrepancies and the reasons.

    • If the prognostic model contained continuous baseline PRO risk predictors, were the ranges valid?

    • Did the authors validate the developed prognostic model or a simplified version of it?

  8. Discussion/Conclusions

    • Did the authors discuss the PRO findings and their prognostic value on OS?

    • Did the authors compare the model to another existing model?


  • OS, overall survival; PRO, patient-reported outcome.

We confirm that we have already started articles screening (January 2018) and we are currently validating the studies resulted eligible for inclusion. We plan to complete data extraction by the end of 2018.

Methodological quality assessment

Critical appraisal of methodological quality and risk of bias of included papers will be undertaken independently by two or more reviewers according to an adapted version of the CHARMS checklist for critical appraisal of prognostic model studies.31 Following these guidelines, the reviewers will extract from individual studies relevant items belonging to the following domains: source of data, participants, outcome to be predicted, candidate predictors, sample size, missing data, model development, model evaluation and results (see box 1).

For each criterion, judgement will be made using one of the following three categories:

  1. Low risk: if the criterion is adequately fulfilled in the study, that is, the study is at a low risk of bias for the given criterion.

  2. High risk: if the criterion is not fulfilled in the study, that is, the study is at high risk of bias for the given criterion.

  3. Unclear: if the study report does not provide sufficient information to allow for a clear judgement or if the risk of bias is unknown for one of the criteria listed above.

Presenting and reporting results

The selection process, including search results and the reason for exclusion, will be reported in a flow diagram, as recommended by the PRISMA statement.33

Data obtained from each eligible study will be summarised by providing descriptive tables reporting authors’ names, publication year, study sample, type of cancer, instrument used to measure PRO and the main methodological findings (eg, multivariable model used, selection procedure, type of model validation, missing data information and model performance).

All relevant issues related to the methodological quality will be carefully reported and described separately. Results from the risk of bias assessment will be presented as counts and percentages, highlighting the most critical domains affecting the overall risk of bias.

Patient and public involvement

No patients will be involved in this study.

Ethics and dissemination

Given that this systematic review will use only published data, ethical permissions will not be required. Findings from this review will be published in peer-reviewed scientific journals and presented at international conferences relevant to various research fields of cancer outcomes research, biostatistics and QoL.


The risk of bias and potential usefulness of prediction models can be adequately assessed only by clear and full reporting of information, about all steps regarding their development and validation. Well-conducted and reported prognostic model studies with a PRO predictor have great potential to improve healthcare delivery, for example, by improving the management of patients in clinical practice and by informing a more accurate stratification of patients in clinical trials. Our results will contribute to identifying key areas of improvement for conducting these analyses and will lay the groundwork for the development of evidence-based guidelines to improve the quality of research in this field.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
View Abstract


  • Contributors FE: conceived this systematic review and provided mentorship and supervision to ND. All authors contributed to the study design leading to the development of this study protocol. ND, FC and FE: wrote the first draft of the protocol, coordinated and integrated comments from all coauthors. FC, FE, GSC and AA: critically revised and edited successive drafts of the manuscript and gave input to the final draft of the protocol, providing methodological guidance on the overall development of the protocol. All authors read and approved the final version of the manuscript.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.