Article Text

Download PDFPDF

Original research
Developing quality indicators for cancer hospitals in China: a national modified Delphi process
  1. Meicen Liu1,
  2. Qingyuan YU2,
  3. Yuanli Liu2
  1. 1 National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China
  2. 2 School of Health Policy and Management, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China
  1. Correspondence to Professor Yuanli Liu; liuyuanli_pumc{at}


Objective Although demand and supply of cancer care have been rapidly increasing in recent decades, there is a lack of systemic quality measurement for cancer hospitals in China. This study aimed to develop a set of core indicators for measuring quality of care for cancer hospitals in China.

Design The development of quality indicators was based on a literature review and a two-round modified Delphi survey. The theoretical framework and initial indicators were identified through the comprehensive literature review, and the selection of quality indicators relied on experts’ consensus on the importance and feasibility of indicators by the modified Delphi process. In addition, indicator weight was identified using the analytical hierarchical process method and percentage weight method.

Setting and participants A panel of leading experts including oncologists, cancer care nurses, quality management experts from various regions of China were invited to participate in the two-round modified Delphi process from October to December 2020. A total of 25 experts completed the two-round modified Delphi process.

Results The experts reached consensus on a set of 47 indicators, comprising 17 structure indicators, 19 process indicators and 11 outcome indicators. Experts gave much higher weight to outcome indicators (accounting for 53.96% relative weight) than to structure (16.34%) and process (29.70%) indicators. In addition, experts also showed concerns and gave suggestions on data availability of specific outcome indicators.

Conclusions Drawing on the comprehensive literature review and the modified Delphi process, this study developed a core set of quality indicators that can be used to evaluate quality performance of cancer hospitals. This is helpful in supporting quality cancer care in China and will provide new insights into the systemic measurement of cancer care internationally.

  • Quality in health care
  • Hospitals, Public
  • China

Data availability statement

Data are available on reasonable request. The data generated during the current study are available from the corresponding author on reasonable request.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • This study adopted scientific framework and rigorous Delphi method to develop the core set of quality indicators and took international evidence-based indicators and features of China’s health system into consideration.

  • The panel of participants comprised top experts in the field of cancer care and quality measurement, from different regions across China, with various expert backgrounds, and had a high response rate.

  • The quality indicators developed in this study are not universal because they are set in the specific health system and social context, thus the practical application in some developing regions of China or other countries may be adopted partly or optimised accordingly.


Cancer is the second-leading cause of global deaths and a vital public health issue worldwide with an estimated 19.3 million new cases and 10.0 million cancer deaths in 2020.1 Rapid population ageing and change in behavioural patterns in conjunction with a large population base in China led to the accumulation of cancer patients in the last decade.2 Currently, approximately 24% of global cancer cases and 30% of cancer deaths happen in China.3 Challenges for resource-limited cancer care services in China are gradually growing. Evidence from China Health Statistics Years between 2010 and 2020 showed that the number of annual patient admissions in cancer hospitals increased by nearly two times, from 1.10 million to 3.24 million, while the number of cancer hospitals increased by 40%, from 111 to 156.4 These data indicate the potential challenges of high workload, intensive service and accelerated efficiency for cancer hospitals in China. Results from China National Health Services Surveys suggested that cancer patients’ willingness to be treated notably increased from 2008 to 2018, indicating emerging latent service demand.5 6 China is also experiencing a transition in cancer profiles towards developed countries, which brings about new challenges for cancer care.7

These amplified and complex needs of cancer care as well as the finiteness of service resources have provoked a primary concern regarding the quality of cancer care. Medical malpractice or medical errors have been widely reported in health settings both worldwide and in China.8–13 Tertiary hospitals treat a large proportion of patients with severe and complicated diseases and offer a relatively high technical level of diagnoses and treatments, as a part of a three-tier health system in China (designated as primary, secondary or tertiary institutions).14 However, questioning cancer care in tertiary cancer hospitals has sprung up in recent years,13 and patient safety-related events have drawn attention to the efforts on measuring quality and improving cancer care in China.

It is well known that developing quality metrics in health settings is essential for continuous quality improvement, which provides national decision-makers and stakeholders with opportunities to benchmark and catalyse improvements. Many international departments and societies have been conducting quality benchmarking and reporting for decades.15–19 The widely used definition of quality of care, formulated by Donabedian, emphasised the quality of technical care (effectiveness, judged in comparison with the best practice) and art of medicine (individual requirements, including convenience, comfort, quietness and privacy).20 21 Donabedian also developed a classic framework for quality assessment, which involved structure, process and outcome (SPO).20 Another important concept related to the quality of care was developed by the Institute of Medicine (IOM) in 2001 (see Crossing the Quality Chasm: A New Health System for the 21st Century), and it put forth six specific aims for improvement: healthcare should be safe, effective, patient-centred, timely, efficient and equitable.22 Based on the universal definitions and frameworks for quality of care, quality metrics in the field of cancer care have been developed in many countries, including the USA, Europe, Canada and Japan.19 23–27 These contents have offered an important theoretical and methodological basis for the development of quality metrics for cancer hospitals in China.

The latest national guideline on the regulation and supervision of cancer hospitals in China was published in 2011, and the latest national guideline for all tertiary hospitals was updated in 2021. A tool for measuring and benchmarking quality of care in tertiary cancer hospitals is required, owing to the emergence of new cancer profiles and therapies as well as some new principles of value care (eg, patient-centred care, integrated care and value in healthcare).28 29 Although some academic studies have developed quality metrics of cancer care for specific cancers in China, systematic quality measures of care in cancer hospitals are lacking.30 31

This study aimed to develop a set of core indicators to measure the quality of care in tertiary cancer hospitals in China by using the SPO framework and conducting a national modified Delphi process. The findings will make an important contribution to the field of benchmark and improvement of quality of care in cancer hospitals in China and will provide new insights into the systemic measurement of cancer care internationally.


Review of concepts, models and indicators

To synthesise concepts, models and indicators of quality measures of cancer care, a systematic review was conducted by searching databases (PubMed, MEDLINE, EMBASE, PROQUEST, Web of Science, CNKI, Wanfang, VIP and SinoMed) and domestic and international office websites related with hospital quality control. We searched databases for studies published between January 2009 and October 2019. The keywords of quality measure were: “quality of health care” or “quality of care” or “quality assurance” or “quality measure” or “quality assessment” or “quality evaluation” or “quality indicator” or “quality metric” or “certification” or “benchmark”and key words of cancer were: “cancer care facilities” or “cancer center” or “cancer hospital”. Our search strategies were listed (online supplemental tables 1,2). We summarised the concepts and classic models of quality assessment, which helped us to structure our indicators. In addition, quality indicators were proposed from the eligible studies and websites that reported at least one SPO indicator related to the quality of care of cancer institutions. The tumour-specific indicators were excluded, for example, proportion of patients with breast cancer receiving immediate reconstruction.

Supplemental material

A total of 106 indicators were extracted from the literature and office websites. First, we compiled these indicators in Chinese including their name, concept and objective, and formulations. Some indicators from western countries were not appropriate and available in China so we revised them in the context of China. Then indicators with repeated formulations or descriptions were merged. The processes of revision and merging were both conducted under the discussions among research team members and Chinese quality improvement experts. Finally 63 indicators were reserved for Delphi consultation after removing repeated indicators (figure 1). In addition, this study designed three-tier evaluation system. The primary indicators were defined as SPO indicators according to the Donabedian’s quality framework.20 The second indicators were defined based on classification of proposed indicators and referred to quality dimensions from literature reviews. The 63 tertiary indicators were allotted into the primary and secondary tiers accordingly.

Figure 1

Flow chart of modified Delphi process.

Selection of expert panel

The selection of an appropriate expert panel is a key step towards the success of a modified Delphi method. Our criteria for the selection of the expert panel were as follows: (1) ample experience and knowledge in the field of hospital management, healthcare quality research and health services; (2) clinical, academic or political background (researchers, clinical oncologists, cancer care nurses, hospital administrators and policy-makers); (3) more than 5 years of work experience in quality measure or hospital administration and (4) full willingness to participate in this study. In order to enhance the application of the measurement tool, we selected experts from various geographical areas and economic zones of China. Experts were invited based on purposive sampling approach from well-known universities, tertiary hospitals and provincial health sectors. Finally, 26 experts were recruited covering seven provinces and four economic zones of China.

Two-round modified Delphi process

Round 1

The Delphi questionnaire for round 1 was designed to collect the experts’ opinions on proposed indicators. The round 1 questionnaire comprised a preface, which demonstrated the study background and objective, and an appendix, which demonstrated the concept, formula and data source of indicators. The three main parts of the round 1 questionnaire were as follows: (1) expert’s basic information; (2) experts’ estimation of the importance and feasibility of indicators and (3) expert’s familiarity with indicators. The indicator importance was scored on a 9-point scale, with higher scores representing higher importance. The indicator feasibility was estimated with ‘yes’ or ‘no’. If the experts considered that a new indicator should be added or they wanted to express any opinions, there was a designed blank area at the end of the questionnaire where they could write down their comments. In the first round, we gave out 26 Delphi questionnaires in Microsoft Word format and asked the experts for their feedback within 2 weeks. An expert refused to participant due to busy work.

Round 2

The indicators were screened following the experts’ feedback from round 1. Indicators were removed, modified or added based on criteria set in advance and experts’ comments. Then, the Delphi questionnaire for round 2 was designed and contained the following two main parts: (1) experts’ estimation of the importance and feasibility of the updated indicators and (2) results and reasons for removing or modifying the indicators. The experts were again asked to estimate indicator importance with a 9-point scale and indicator feasibility with ‘yes/no’. Any comments were welcomed. In addition, arithmetic mean of indicator importance, frequency of indicator feasibility and expert’s own scores at the round 1 were returned to experts when they completed round 2.

Criteria for screening indicators

To screen indicators, three key statistics of indicator importance (arithmetic mean, coefficient of variation and frequency of ranking high score of ‘7–9’) were used. The experts’ estimation of indicator feasibility was used as a reference to determine the difficulty degree of data acquisition.

Many studies have reported a definition of consensus with a priority criterion, and a few studies have reported a post hoc criterion for exclusion when the indicator’s rank is not well discriminated.32 In our study, a priority criterion with fixed cut-off values had been set before two-round screenings were conducted (table 1). In addition, a boundary value criterion was used for a post hoc exploratory analysis, where cut-off values were altered with data distribution of every round’s feedback.33 Comparing the priority criterion with the boundary value criterion, we found that the boundary value criterion was looser for the first-round screen and stringent for the second-round screen (table 1).

Table 1

Comparison between the priority criterion and the boundary value criterion

The priority criterion with fixed cut-off values

The priority criterion to screen indicators was based on the following three conditions:

  1. The arithmetic mean for indicator importance ≥7.0.

  2. The coefficient of variation for indicator importance ≤0.20.

  3. The frequency of ranking high score of ‘7–9’ for indicator importance ≥70%.

An indicator was removed if none of the three conditions were met.

The boundary value criterion

For arithmetic mean and frequency of ranking high score of ‘7–9’, the boundary value was set at ‘mean−SD’; for coefficient of variation, the boundary value was set at ‘mean+SD’. An indicator was removed if none of the three conditions were met.

Experts’ comments

We summarised experts’ comments for each indicator at the two-round Delphi process. The comments showed experts’ specific suggestions on each indicator, including expression, formula, data acquisition, similarity and collision of indicators. These comments were used to modify indicators and deepen our understanding and discussions on these indicators.

Data analysis

We analysed the experts’ positive coefficient, authority coefficient and coordination coefficient. Higher positive coefficient, authority coefficient and coordination coefficient indicate better credibility of results. The positive coefficient reflects the experts’ positive input and effective response rate. The coordination coefficient reflects the consistency of the experts’ evaluation. Kendall’s W concordance coefficient reflects the coordination of the experts’ scores of indicator importance.

The authority coefficient measures the experts’ knowledge and degree of authority for the indicators evaluated. Experts’ authority coefficient (Cr) was calculated by the level of expert’s familiarity (Cs) and the judgement coefficient (Ca). The Cs was measured on a 5-point Likert scale (unfamiliar, less familiar, average, more familiar and very familiar, defined as 0.2, 0.4, 0.6, 0.8 and 1 point, respectively). The Ca was measured according to experts’ judgement criteria (table 2). The judgement criterion has widely been used in modified Delphi studies.33 34 Generally speaking, Cr≥0.7 was considered acceptable. The level of authority (Cr) was calculated as follows:

Embedded Image

Table 2

Judgement criterion

Indicator weight calculation

We calculated the weight of primary and secondary indicators using the analytical hierarchy process (AHP) method, and that of tertiary indicators using the percentage weight method.33 35 In the AHP method, Saaty’s 9-point scale was adopted to identify the relative importance of the indicators. The AHP method can measure the consensus of the experts’ judgements by calculating a consistency ratio (CR), and CR<0.10 is considered as acceptable.36 CRs in our study were all below 0.05, at a sound acceptable level (online supplemental table 3). The Yaahp software V.10.0 (Yaahp software, Taiyuan, Shanxi, China) was used for the AHP method, and the Excel software was used for the percentage weight method.

Reporting of this research adheres to the COnsolidated criteria for REporting Qualitative research guidelines.

Patient and public involvement

Patients were not involved in this study.


Delphi experts’ basic information

A total of 25 experts completed the two-round modified Delphi process (table 3). They were experts in quality assurance and assessment and worked in different regions in China (Beijing, Shanghai, Sichuan, Hubei, Heilongjiang, Jiangsu and Chongqing). They were from different backgrounds and represented different stakeholders. Namely, 17 experts were affiliated with clinical departments, nursing departments, hospital-acquired infection control departments and quality control departments of hospitals; four experts were affiliated with universities; and four experts were affiliated with national or regional health commission departments.

Table 3

Delphi experts’ basic information

Delphi experts’ positive, authority and coordination coefficient

In the first round, the Delphi questionnaires were sent to 26 experts. A total of 25 (96.2%) experts completed it, and 44% of those put forward comments for quality indicators. In the second round, the Delphi questionnaires were sent to 25 experts who responded in the first round. All of the 25 experts completed it, and 12% of them put forward comments.

Ca, Cs and Cr were 0.782, 0.928 and 0.855, respectively, in the first round, and 0.760, 0.928 and 0.844, respectively, in the second round. The level of authority of the modified Delphi process was high, and our results had a good reliability.

Our results also showed that experts’ scores on secondary and tertiary indicators were consistent (all p<0.001). The experts’ scores on primary indicators reached consistency in round 2 (p<0.05). In addition, all Kendall’s W concordance coefficients increased from the first round to the second round, implying improved consistency (online supplemental table 4).

The results of the modified Delphi process

Indicator screening

According to the experts’ comments and indicator screening based on the priority criteria, 15 tertiary indicators were removed, 3 secondary indicators were modified and 4 new tertiary indicators were added in the first round; 2 tertiary indicators were deleted and 3 newly added tertiary indicators were combined with the previous indicators in the second round (online supplemental table 5). Finally, a total of 15 secondary indicators and 47 tertiary indicators were reserved. The secondary indicators comprised 4 structure indicators, 6 process indicators and 5 outcome indicators; the tertiary indicators comprised 17 structure indicators, 19 process indicators and 11 outcome indicators. The set of quality indicators that reached consensus are shown in table 4.

Table 4

Indicators of quality of care in Chinese tertiary cancer hospitals

In comparison with the indicator screening based on the priority criteria, the secondary indicators were consistent but only 43 tertiary indicators were reserved using the boundary value criteria. The tertiary indicators comprised 16 structure indicators, 16 process indicators and 11 outcome indicators (table 4).

Indicator feasibility

We collected the feasibility of quality indicators and found that the feasibility of most of the tertiary indicators was more than 90% (table 4). The experts expressed concerns about the difficulty of availability of some indicators, especially for 5-year survival rate, patient’s improved health status and function recovery, although they thought these indicators were essential for the quality of cancer care. The 5-year survival rate at hospital level was unavailable for many hospitals due to the lack of long-term follow-up. There were no consistent measuring scales of patient health status (such as a patient-reported outcome measure tool) used for all Chinese cancer hospitals. The experts commented that these two indicators’ (5-year survival rate, patient’s improved health status and function recovery) measuring and data collection were challenging. Efforts on setting up long-term follow-up cohorts for Chinese tertiary hospitals have been made, and the nationwide patient-reported outcome measure at hospital level has recently also received growing attention in China. The feasibility of these indicators may be appropriately solved in the future.

The weight of quality indicators

Based on the AHP method and the percentage weight method, indicator weights were calculated (table 4). The primary indicator weights for structure indicators, process indicators and outcome indicators were 0.1634, 0.2970 and 0.5396, respectively.

Structural indicators

The weight of structural indicators accounted for 16.34% of all indicators. Among the structure domains, information system indicator had the largest proportion of weight (0.0737), followed by structure of staff (0.0426), research and training (0.0276) and service capacity (0.0194).

Process indicators

The weight of process indicators accounted for 29.70% of all indicators. Among the process domains, the indicator weights were as follows: treatment quality (0.0918), diagnosis quality (0.0729), rational administration of drug (0.0579), symptom management (0.0334), follow-up and education (0.0210) and process management (0.0201).

Outcome indicators

The weight of outcome indicators accounted for the largest part of all indicators, with a 53.96% proportion. Among secondary indicators of the outcome domain, the weight of patient safety (0.2106) was the largest, followed by that of the treatment outcome (0.1526), hospital efficiency (0.0770), patient satisfaction (0.0582) and medical expense (0.0412). Regarding tertiary indicators, the five most important indicators were 5-year survival rate (0.0782), patient’s improved health status and function recovery (0.0744), patient satisfaction (0.0582), mortality of low-risk group (0.0432) and unplanned event rate within 30 days (including readmission, return to intensive care unit, and return to operation room) (0.0432).


Quality indicators and weight

Traditionally, the SPO framework is generally adopted for constructing an indicator system, which preferably covers the contents of resources needed, service process and health benefits that patients preserve through receiving services.37–39 In this study, the SPO was used as the primary indicator for categorising quality indicators of cancer care. Regarding indicator weight, outcomes indicators had the largest weight, while structure indicators had the lowest weight in this study. Outcome indicators, with a weight of over a half, reflected the experts’ substantial emphasis on patients’ health benefits from received care. Up to date, the grade certification of cancer hospitals in China mainly assesses the resources that they own and service processes, which is different from experts’ perspectives on quality cancer care. From an international perspective, outcome indicators have an important position in the quality evaluation of healthcare as well.40 For structure and process domains, there were more indicators included because of their easiness to monitor and access in China. In addition, based on the boundary value criteria, a core set of indicators was reserved, which could be used when resource is limited or collecting too many indicators is impracticable.

Outcome indicators covered five aspects, as the secondary indicators showed, namely patient safety indicators, treatment outcome indicators associated with survival, improved health status and recovery, patient satisfaction indicators, and efficiency and cost indicators. This design of secondary indicators for cancer care outcome basically aligned with the six dimensions IOM proposed in 2001.22 Patient safety mainly involved mortality and adverse events. Patient satisfaction with health providers’ services is an emerging source of hospital evaluation, where information is collected from patients’ perspectives and it embodies the principle of patient-centredness.30 Efficiency and cost indicators reflect the service’s accessibility and affordability to some extent. The extremely vital indicators in cancer care evaluation are survival, improved health status and recovery, where a longer survival time and a quicker recovery period are the ultimate goals of care.41.

Process indicators cover the entire streamline of diagnosis, treatment and rehabilitation. Experts place more emphasis on the diagnostic accuracy and conformity of treatment guidelines. In particular, multidisciplinary diagnosis and treatment has been recognised as a priority for patients with cancer, which has been widely referred to and adopted both worldwide and in China.29 42 43 In addition, care comfort and symptom alleviation are recognised as necessities, which represent patient-centred principles through showing concerns and supporting considerable care for patients’ pain, sleep disturbance, malnutrition and depression. Structure indicators involve four aspects, namely staff, information system, service capability, and research and training. A superior information system could help simplify workflow and enhance service efficiency. The use of intelligent medical decision support systems, such as treatment checks and reminders, drug monitoring, and clinical path recommendation, may help decrease the occurrence of medical errors.

Overall, this set of quality indicators developed in China had same framework and similar measuring aspects with the international quality systems. A umbrella review summarised hospital performance indicators globally.44 This review found over half of studies covered efficient, effective, patient-centred and safe aspects, and some studies covered responsive governance, staff orientation and timeliness. The set of quality indicators developed in this study is lacked of indicators related with responsive governance, which refers to that hospitals respond to community needs and ensure the continuity and coordination of services. In addition, compared with quality indicators developed in OECI BENCH-CAN project, this study had little consideration in organisational governance and leadership.23 This may indicate the overlook for governance among hospital administrators and researchers in China. The optimal way to measure governance and leadership in China is currently lacking. For tertiary indicators, diversity existed compared with international measures. Indicator of adverse events reporting system was adopted in this study, while some studies developed tools to identify and monitor patient adverse events from patient’s or staff’s perspectives, such as CTCAE and PRO-CTCAE.45 46 These novel ways remain to be tested in China in the future.

Indicator feasibility and practical application

Some experts expressed concerns about the feasibility of two indicators (survival rate and improved health status and function recovery) in their comments during the modified Delphi process. These experts also claimed that survival rate was important for cancer care evaluation, but its utilisation should be cautious for the following reasons. First, many hospitals cannot calculate this indicator due to the lack of long-term follow-up, especially for those in less developed areas. Second, patients may receive treatments at various hospitals; thus, the survival rate does not necessarily reflect the effects of care from only one hospital. When survival rate is used for hospital quality evaluation, patients receiving treatments at various hospitals should not be enrolled. Third, if there are too few patients in one hospital, the estimation of the survival rate is not robust and may cause misleading evaluation. An expert discussing whether survival statistics make sense for every hospital expressed similar concerns.47 International experts also emphasised that outcome indicators are vulnerable to inadequate sample size and statistical power.41 48 For these reasons, the survival rate is claimed to be optional when applying this set of quality indicators. Regarding improved health status and function recovery indicator, different measuring scales across hospitals and measuring frequency at least two times bring challenges for their data collection and application. In addition, the variability of electronic medical reporting systems at various regions of China will impact the practical implementation of quality indicators. According to the National Health Commission statistics, 232 of 2817 tertiary hospitals were rated 5 and over in the grades system of electronic medical records(grades ranged 1–8).

Future work and challenges

Constructing a set of indicators for measuring the quality of care in cancer hospitals is an essential step for quality evaluation and improvement; however, future work needs to bridge the gap between tool construction and practical application. The validation and practical application for this tool are conducting at about 30 tertiary cancer hospitals across China. The data acquisition and preparatory data analyses are ongoing. Some important points are as follows:

Empirical validation of tools

Empirical studies should be conducted to test the validity and reliability of quality measures. Although every indicator of quality of care is identified based on previous studies, it remains to be verified whether a composite index after indicator scoring and weighting could appropriately discriminate a hospital’s care quality.49 50

Risk adjustment for indicators

The indicator set developed in our study helps to measure the quality of care and conduct comparisons across hospitals or across different periods. When conducting a comparison, risk adjustment is a prerequisite, as otherwise underestimation of performance for these practices serving vulnerable patients is possible.51 52 Previous studies have also found that risk adjustment played a larger role for outcome indicators than for process indicators, indicating the importance of casemix adjustment for outcome indicators.53

Public reporting of results

Publicly reported hospital performance enhances transparency, spurs quality improvement and empowers patient choice.54 For policy-makers and hospital governors, quality measuring and reporting could help target primary improvement points. Patients also have increasing enthusiasm for public reporting of hospital performance so that they can seek proper and better care.55 Ensuring that reports are clear, precise and understandable is also a critical step for the profound impact of hospital evaluation. League tables and funnel plots are applied to this end.56

Strengths and limitations

This study has some strengths. First, this study adapted scientific framework and method, and took both international evidence-based indicators and features of China’s health system into consideration. Second, the panel of experts comprised people from different regions across China, represented various expert backgrounds and had a high response rate. Third, this study provided new insights into the systemic measurement of cancer care from Chinese experts’ perspectives.

This study also has some limitations. First, the entire application of this tool is limited in other countries because of different health systems and social contexts. Second, some indicators with compromised feasibility were reserved because of their significance, but their practical application in some developing regions in China may be adopted partly or optimised accordingly. Third, patients’ viewpoints are not collected, therefore, it is lacking that how important these quality indicators are from a patient perspective. In addition, although this study tried to cover various regions across China, most experts included were from the Eastern area of China.


This was a national expert consensus study for developing a set of quality indicators to measure the quality of care in tertiary cancer hospitals in China. Drawing on a comprehensive literature review and the modified Delphi process, a set of 47 indicators were identified, comprising 17 structure indicators, 19 process indicators and 11 outcome indicators. A panel of Chinese leading experts gave much higher relative weight to outcome indicators (accounting for 53.96% relative weight) than to structure (16.34%) and process (29.70%) measures, and they also showed modest concerns about data availability issues of outcome indicators. This set of indicators could practically quantify the quality of care in tertiary cancer hospitals in China and provide policy decision-makers and hospital governors with opportunities to benchmark and catalyse improvements. More work will be done in the next step including empirical validation of quality indicators and optimisation of data acquisition method for particular indicators.

Data availability statement

Data are available on reasonable request. The data generated during the current study are available from the corresponding author on reasonable request.

Ethics statements

Patient consent for publication

Ethics approval

This study involves human participants and this study was approved by the Ethics Committee of Institute of Medical Biology of Chinese Academy of Medical Sciences (IPB-2020-23) and all Delphi experts provided informed consent for their participation in the study. Participants gave informed consent to participate in the study before taking part.


We are grateful to the experts who participated in the modified Delphi process. We thank LetPub ( for its linguistic assistance during the preparation of this manuscript.



  • Contributors ML: conceptualisation, methodology, software, data curation, investigation, visualisation and writing—original draft. QY: conceptualisation, supervision and writing—review and editing. YL: supervision and writing—review and editing. All authors read and approved the final manuscript before submission. ML serves as a guarantor for the overall content of this publication and ensures the accuracy and integrity of this research.

  • Funding This work was supported by the Chinese Academy of Medical Sciences Innovation Fund for Medical Sciences(2021-I2M-1-046).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.