Article Text
PDF
Abstract
Objective This study aims to advance understanding of globally valid versus country-specific quality dimensions and indicators, as perceived by relevant stakeholders. It specifically addresses patient-level indicators for cataract surgery.
Design A mixed-methods case study comparing Singapore and The Netherlands
Setting Singapore (2017–2019) and The Netherlands (2014–2015).
Participants Stakeholder representatives of cataract care in Singapore and The Netherlands.
Intervention Based on the previously identified complete set of stakeholders in The Netherlands, we identified stakeholders of cataract care in Singapore. Stakeholder representatives then established a multi-stakeholder perspective on the quality of cataract care using a concept mapping approach. This yielded a multidimensional cluster map based on multivariate statistical analyses. Consensus-based quality dimensions were subsequently defined during a plenary session. Thereafter, Singaporean dimensions were matched with dimensions obtained in The Netherlands to identify commonalities and differences.
Main outcome measure Health-services quality dimensions of cataract care.
Results 19 Singaporean stakeholders representing patients, general practitioners, ophthalmologists, nurses, care providers, researchers and clinical auditors defined health-services quality of cataract care using the following eight dimensions: clinical outcome, patient outcomes, surgical process, surgical safety, patient experience, access, cost and standards of care. Compared with the Dutch results, 61% of the indicators were allocated to dimensions of comparable names and compositions. Considerable differences also existed in the composition of some dimensions and the importance attached to indicators.
Conclusions and relevance This study on cataract care in Singapore and The Netherlands shows that cataract care quality measurement instruments can share a common international core. At the same time, it emphasises the importance of taking a country-specific multi-stakeholder approach to quality definition and measurement. Complementing an international core set with country-specific measures is required to ensure that the included dimensions and indicators adequately capture the country-specific quality views.
- health policy
- international health services
- quality in health care
- cataract and refractive surgery
Data availability statement
Data are available upon reasonable request. The data from the concept map respondents are stored in the concept mapping software and can only be accessed with the password of the authors. The authors are available to provide such access if requested. All subsequent data are included in the supplementary materials.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The study presents a multi-stakeholder perspective which includes all salient stakeholders.
The study combines perspectives from the distinct leading health systems of Singapore and The Netherlands.
Using concept mapping, the studies combine quantitative and qualitative techniques to present consensus-based quality dimensions.
Cataract care is a highly standardised mature health service for which quality measures are well understood, and it therefore serves as a robust case study to identify differences in quality perspectives between stakeholders and countries.
The methods are time consuming for participants, which causes some time pressure on the data collection and consensus building process.
Introduction
Standardised measures are important to measure, monitor, analyse and improve the quality of health service delivery. The International Consortium for Health Outcomes Measurement (ICHOM) proposes global minimum sets of outcome measurements for health services to standardise outcomes and improve processes globally.1 While having received much recognition, the value of the ICHOM sets has also been debated. The implementation of such international standards remains a major challenge, for instance, for the globally most common surgical procedure cataract.2 While the use of large electronic registries allows for large-scale tracking,3 adherence to the proposed standardised sets is limited.2 Currently, the outcome measures of cataract surgery vary across countries and hospitals.2
ICHOM characterises the proposed global set for cataract care as a compromise between the usefulness of data and the practicalities of data collection.1 The set is developed using a Delphi method. The Delphi panel, however, may not have fully included all salient stakeholders as it predominantly consisted of ophthalmologists while failing to represent, among others, health insurance providers and policymakers.4 Moreover, it remains unclear whether country-specific characteristics are appropriately accommodated, reducing the validity as perceived by local stakeholders. This is especially relevant as quality definitions and dimensions are evidenced to vary across countries.5 6
This study aims to advance understanding of globally valid versus country-specific quality dimensions and indicators, as perceived by all relevant stakeholders. It focuses on patient level dimensions and indicators and has engaged and involved patients as an important stakeholder, ensuring that their needs and preferences are included, in alignment with the principles of people-centred health services.7 We conducted a case study comparing cataract surgery between Singapore and The Netherlands. The Netherlands has topped the rankings of the European Health Consumer Index from 2008 to 2016 and can be viewed as a leading representative of a Western healthcare system.8 9 Singapore’s health system is similarly considered as leading and has been identified as the best performing health system outside of Europe by the WHO.8
The high quality health systems of both countries provide accessible cataract care. In The Netherlands, all citizens are mandated to purchase statutory health insurance from private insurers which covers cataract surgery.10 In Singapore, the reimbursement system is anchored in the twin philosophies of individual responsibility and affordable healthcare.11 Singaporean patients are required to provide a copayment for cataract surgery of approximately 30% from their medical savings account.12 In addition to health system differences, the organisational cultures and attitudes towards health also differ essentially.13 14
Cataract surgery is one of the most cost-effective and frequently performed surgical procedures worldwide, as cataract is still a leading cause of blindness globally.15 The resulting importance of advancing a comprehensive understanding of quality measures for cataract care has already motivated the development of several global registries.16–20
Methods
Study design
We conducted a concept mapping study between 2017 and 2019 in Singapore to define quality dimensions of cataract surgery and to systematically compare results with those obtained in The Netherlands between 2014 and 2015.4 Below, we present the Singaporean study process and the methods used to identify the commonalities and differences between the quality dimensions of the two countries. We begin with a brief description of concept mapping and a summary of the Dutch data and results.4 21
Written informed consent obtained from all participants prior to participation. Participants were reimbursed for their time and travel costs.
Concept mapping
Concept mapping is a structured group conceptualisation designed to integrate input from multiple stakeholders with different expertise or interests on a set of items. It results in a visualised clustering of the set of items which represents the integrated input.12 13 Concept mapping is a well-defined and reproducible mixed method that allows for both qualitative and quantitative comparisons, which is a relative strength over other approaches such as Delphi studies.22–25 Through its participatory nature, it combines group processes with multivariate statistical analyses. There is no strict limit to the number of participants that should be involved in concept mapping, although the inclusion of 10–20 participants is advised.22 We invited participants representing all relevant stakeholders following a stakeholder theory-based protocol,4 26 while ensuring equivalence between stakeholders of the two countries as much as possible. To include all relevant stakeholders of cataract care in Singapore, we initially selected a Singaporean counterpart for each of the stakeholders included in the Dutch study, and then subsequently added stakeholders considered relevant by the researchers or stakeholders already included. Next, representatives of all identified stakeholders were invited to participate in our study.
For this study, quality indicators for cataract care formed the items of interest. The set of items was obtained by combining all indicators included in sets obtained through systematic search of scientific and grey literature and allowing researchers and stakeholders to add or delete items in case of consensus, as described in.4 This list involved health service quality indicators relevant at the patient level. Quality indicators at the population or national level, such as those included in the global action plan of the WHO, were excluded.27
Following the concept mapping methods, each participant individually sorted the items into groups according to similarity, and then labelled each pile and rated the importance of each item on a 5-point Likert scale. These data were analysed using Concept Systems Global MAX,28 which uses multivariate statistical analyses and hierarchical clustering.23 The resulting clusterings and maps were interpreted by participants in group discussions, reaching consensus on a minimal number of well-defined clusters for which cluster labels were agreed.
Preceding study in The Netherlands
After a systematic inclusion process, the following stakeholders were included in the Dutch study4: patients, the patient federation, ophthalmologists, general practitioners, optometrists, hospitals, private clinics, health insurers and the national healthcare institute (which represented the government), see table 1. Dutch data were collected in 2014–2015. The resulting consensus-based clustering into quality dimensions in The Netherlands can be found in table 2.
Participation of stakeholder representatives in The Netherlands and Singapore
Dutch and Singaporean multidimensional scaling maps
Study process in Singapore
To include all relevant stakeholders of cataract care in Singapore, we initially selected a Singaporean counterpart for each of the stakeholders included in the Dutch study, and subsequently added stakeholders considered relevant by the researchers or stakeholders already included. This resulted in the inclusion of patients from Chinese, Indian and Malaysian origin. Next, representatives of all identified stakeholders were invited to participate in our study. Further details are provided in the results section.
Between-country comparison
We used a descriptive approach for the cross-country comparison between Singapore and The Netherlands.21 First, we listed all items per cluster for Singapore and The Netherlands. Second, we compared clusters between the two countries and matched clusters based on item and label commonality. More specifically, we first calculated the number of items in common for each pair of clusters from Singapore and The Netherlands. Next, we formed cluster pairs consisting of one Singaporean and one Dutch cluster, with the objective to maximise the sum of the numbers of items that these paired clusters had in common, while also taking cluster labels into account. For the resulting pairs, we then described similarities and differences between the two countries regarding the items in the paired clusters and the importance ratings of these items.
Throughout the manuscript, we apply the cluster labels defined by Singaporean participants unless specified otherwise. The interpretation of the similarities and differences is left for the discussion.
Results
Participants in Singapore
The seven stakeholder groups of cataract care in Singapore were represented by 19 participants: patients, general practitioners, ophthalmologists, care providers, optometrists/nurses, researchers in health services and management/auditors. Patients were of Chinese, Indian or Malaysian origin. Out of a total of 19 participants, 14 participants conducted the digital sorting and rating tasks, and 12 attended the group meeting. The Singaporean optometrists and nurses only participated in the group discussion. Table 1 provides an overview detailing the participation of stakeholder representatives at each stage of the study. The researchers were included as representatives of the Singaporean government, which was identified as a salient stakeholder but chose not to participate.
Consensus building among stakeholders in Singapore
Singaporean stakeholder representatives sorted the 125 items into an average of 10 piles (mean (M)=10, SD=4.8) and rated them with a mean importance of M=3.75 (SD=0.38), suggesting a high overall importance of the items. The items and their average importance ratings can be found in online supplemental appendix 1.
Supplemental material
Stakeholder representatives reached consensus during the plenary meeting that the multidimensional scaling (MDS) map with eight clusters provided most meaningful quality dimensions. Considering the clusters clockwise as presented in figure 1, the agreed-upon labels were as follows: clinical outcomes, patient outcomes, surgical process, surgical safety, patient experience, access, costs and standard of care. One participant completed the sorting and rating assignments after the plenary session. These additional data had only very minor effects on the resulting concept map. Subsequent adjustments were communicated to and approved by all participants involved.
Multidimensional scaling (MDS) map for Singapore with eight clusters (matching items with Dutch MDS map in green).
The ‘stress value’ for the final MDS map was 0.29, indicating that the model demonstrated a satisfactory fit (concept maps have an average stress value of 0.2829). The average bridging values per cluster, which are indicative of the relative agreement on rated items, are presented in table 2. These bridging values indicate that stakeholders demonstrated strong agreement on the grouping of outcomes and experiences (clusters D, E, F), and weaker agreement on indicators clustered as standards of care (cluster G).
Between-country comparison
As shown in table 2, the Dutch and Singaporean MDS maps consisted of seven and eight clusters, respectively. Comparing item commonality and labelling between clusters of The Netherlands and Singapore reveals that clusters D through H can be straightforwardly mapped identically (D to D, E to E and so on) between the two countries (see online supplemental appendix 2). The overall matched number of items is maximised by matching B to B and C to C, and leaving the Singaporean cluster A (costs) unmatched. Alternatively, when allowing for one Dutch cluster to be matched to two Singaporean clusters to accommodate the difference in number of clusters, the Singaporean clusters A (costs) and B (access) can be matched to Dutch cluster B (patient centredness and accessibility).
Supplemental material
As a result, 76 of 125 items (61%) are in matched clusters in both countries—85 (68%) when allowing the Singaporean clusters A (costs) and B (access) to be matched to the Dutch cluster B (access). Items in cluster D (patients experience), cluster E (clinical outcomes) and cluster H (surgical process) corresponded most between The Netherlands and Singapore. Items in cluster C (surgical safety) and G (standard of care) corresponded least.
In addition to appropriately matching clusters on items, the matching also resulted in matching labels to corresponding ones. For example: safety versus surgical safety and experienced outcomes versus patient outcomes. The eighth cluster that exists in Singapore but not in The Netherlands is labelled cost (discussed extensively below). Figures 1 and 2 show the MDS maps of Singapore and The Netherlands. To aid visualisation of the similarity between those MDS maps, points that represented the 76 items that were sorted in a corresponding cluster between countries are coloured green.
Multidimensional scaling (MDS) map for the Netherlands with seven clusters (matching items with Singapore MDS map in green).
Stakeholder representatives of both countries rated clusters C (surgical safety) and D (patient experience) as most important. Furthermore, cluster D (patient experience) contains the most items rated in the top-10 in The Netherlands and Singapore.
Discussion
Main findings
This study showed that while health-services quality dimensions and indicators for cataract surgery—as well as their importance—are largely shared between The Netherlands and Singapore according to relevant stakeholders, there are also important differences. We found that considerable inter-country similarities exist in labelling health-service quality dimensions. On the other hand, we found that the resulting dimensions and valuation of the indicators are less uniform between countries. To appreciate the differences, we interpret the results per cluster below, from the least to the highest level of correspondence.
Costs
In The Netherlands, the mandatory health insurance fully covers cataract surgery,10 whereas there is a copayment of 30% in Singapore.12 This may explain why cost is a separate cluster in Singapore but not in The Netherlands. Consequently, many of the items included in the Singaporean cost cluster are found in the patient centredness and accessibility cluster as identified in The Netherlands.
Standard of care
Standard of care is another cluster with much difference between The Netherlands and Singapore. The Dutch nationwide registration, which contains complications, may have reinforced the importance the Dutch attach to standards—a phenomenon not found in Singapore. By contrast, public hospitals in Singapore are subjected to annual patient experience surveys by the Ministry of Health, which is not the case in The Netherlands. This might explain why Singaporean stakeholders sorted related items (eg, items 71, 72, 77, 78, 79) under standard of care, whereas The Dutch sorted them under patient outcomes.
Surgical safety
Surgical safety is found to be of high importance in both countries. Several items clustered under surgical safety by Singaporean stakeholders—for example, the provision of information and choice options for patients (items 29, 38, 44)—are included in the patient experience cluster in The Netherlands. This might be due to cultural variations in which Dutch respondents believe patients should be well informed and engage in shared decision-making on equal terms, whereas the Singaporean patients might believe the ophthalmologist should take responsibility for risks, safety and decision-making.13
Access
The concern for costs in Singapore appears to translate to a limitation of patient choice regarding cataract surgeon, medication and type of intraocular lenses. Freedom of choice implies a higher copayment. The corresponding items related to, for example, the choice of specific cataract surgeons, prescription of medication or the type of intraocular lenses, may therefore be associated with access by Singaporean stakeholders, whereas this was not the case from the Dutch perspective.
Further, Singapore has begun only recently to adopt a model of care already established in The Netherlands. In this model, the role of the ophthalmologist is reduced and the role of others, such as optometrists and nurses, is increased. In The Netherlands, several of the items that relate to such care models (eg, 91, 104, 98, 111) ended up in clusters surgical safety and standard of care, whereas Singaporean stakeholders considered these items in the access cluster.
Patient outcomes
Despite the present discussion on value-based healthcare which emphasises the importance of patient-reported outcome measures (PROMs), the cluster patient outcomes scored fourth out of eight on average in Singapore, and the corresponding cluster scored third out of seven in The Netherlands. The PROM items 65, 66, 67, 68 were rated as relatively important by Dutch stakeholders but less so by the Singaporean stakeholders. Other clusters and indicators, particularly surgical safety and patient experience, were perceived as more important in defining the quality of cataract care.
Patients experience
In terms of average importance score, the patient experience cluster scored highest in both countries. However, patient engagement and patient involvement differ between Singapore and The Netherlands. Communication to patients and information provisioning have been institutionalised in The Netherlands for several years, while bodies such as patient councils in hospitals have not been introduced until recently in Singapore. This may explain why several items related to informing and empowering patients (eg, items 29, 38, 41, 45, 46) are clustered in the patient experience cluster by Dutch stakeholders, while the Singaporean stakeholders sorted them mostly under surgical safety.
Surgical process
The surgical process cluster overlaps by 80% among the two countries. Similar to the clinical outcomes cluster, this is likely because of the technical nature of these items, which are subject to long lasting international discussion.1 3 7 8
Clinical outcomes
The clinical outcomes cluster shows little difference between The Netherlands and Singapore. Clinical outcomes might be relatively easy to compare globally as they are hardly affected by cultural variations across countries. The global consensus may result from the long-lasting international discussion of clinical outcomes via the scientific literature, textbooks, international ophthalmological bodies and organisations like the WAEH (World Association of Eye Hospitals). Moreover, both countries have well-established registries for clinical outcomes which are linked to international registries.
Relationship to previous studies
As previously described,1 the ICHOM cataracts standard set focuses on clinical outcomes, patient-reported outcomes and surgical techniques—indicators which are relatively straightforward to measure. The cluster clinical outcomes covers many of the indicators included in the ICHOM set. Although PROMs are included in the important cluster patient outcomes, in The Netherlands, the PROMs included in the ICHOM were not among the 10 indicators perceived as most important in The Netherlands or Singapore. Stakeholders in our study have selected other patient-related dimensions as more important in defining quality, as for instance related to communication and information provisioning. None of these highly important items are part of the current ICHOM cataracts standard set.
At the country level, our study confirms that differences in cultures and health systems result in differences in quality perspectives and comparability.21 It confirms that existing international standardised sets such as the ICHOM cataracts standard set1 can serve as a basis yet need refinement to adequately capture the quality perspectives of local stakeholders.2 Appreciation of these local perspectives requires rich contextual information and can subsequently translate to country-specific quality dimensions and measures which have broad stakeholder consensus. Indeed, the current study emphasises the importance of blending globally prioritised quality dimensions and indicators with country-specific ones. Moreover, when it comes to practical implementation in a country, it is important to compose an appropriately-sized set of indicators which are reliable and valid in the context of the country, and for which data collection is feasible. Based on stakeholder input and consensus, this set may include additional indicators.4
Strengths and limitations
The benefits of using concept mapping to create consensus across stakeholder perspectives also come with some methodological limitations. Although participants were carefully instructed regarding the approach, participants appeared to have some difficulties with its open-ended nature (eg, the process of labelling clusters). As a result, many questions were raised during the plenary meeting regarding methodology, leaving less time for discussion and interpretation of the MDS map. In future research, more time could be allocated to explaining the theoretical background and method of concept mapping in advance. Moreover, in both countries the government was unwilling to be directly involved. While solutions were found in both countries, the lack of direct government representation is a limitation of our study.
Despite these limitations, our method of concept mapping has advantages over the Delphi method previously used to define a set of quality indicators for cataract care among stakeholders.1 Concept mapping can better synthesise and cope with input from a broad and diverse set of stakeholders. It weighs the individual contributions provided prior to the plenary meeting equally, and subsequently creates consensus on dimensions without having to compromise on differences. Further, it gives quantified and visualised insight into dimensions and the subsequent similarities and differences between countries.
Conclusion
This study shows that while many similarities exist between the identified quality dimensions and their perceived importance in Singapore and The Netherlands, there are also clear differences between the two countries. Together with the differences among stakeholders per country, the findings demonstrate the importance of taking a country-specific multi-stakeholder approach to quality definition and measurement. The implementation of country-specific quality measurement sets can be based on a common international core yet requires identifying country-specific measures to effectively reflect the quality perspectives of local stakeholders.
Data availability statement
Data are available upon reasonable request. The data from the concept map respondents are stored in the concept mapping software and can only be accessed with the password of the authors. The authors are available to provide such access if requested. All subsequent data are included in the supplementary materials.
Ethics statements
Ethics approval
This study was conducted with approval from the SingHealth Institutional Review Board (2016/2649 (R1370/56/2106-2019)).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors AS-V co-designed the research, collected the data, analysed the data and was in the lead of the writing. DDK conceptualised the research, coordinated the data collection in Singapore and contributed to the introduction and discussion section. EL contributed to the data collection in Singapore, and contributed to the introduction and discussion section. CW facilitated the data collection in Singapore, and contributed to the introduction and discussion section. JJVB conceptualised the research, coordinated the data collection in The Netherlands, initiated the data collection in Singapore, supervised parts of the data analysis and critically reviewed the manuscript. JJvdK supervised the research, supervised parts of the data analysis and co-wrote the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement statement Representatives from Patient Organisations and Patients who volunteered to take part in the research provided quantitative and qualitative input to represent the view of the stakeholder patient. They also took part in the stakeholder group discussion in which results were interpreted and consensus on quality dimensions was reached. Both in Singapore and in The Netherlands the patient respondents have approved the national results.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.