Objective To gain information about the advantages/disadvantages of an implementation of patient-reported outcome measures (PROM) into the clinical routine of trauma/orthopaedic surgeons, and to identify the technical constraints confronting a successful implementation of PROMs.
Design Online survey.
Participants Surgeons who are members of the AO Foundation.
Measures Participants answered questions regarding demographics, their familiarity with specific and generic PROMs and the use of PROMs in clinical routine. Furthermore, reasons for/against using PROMs, why not used more often, prerequisites to implement PROMs into clinical routine and whether PROMs would be implemented if adequate tools/technologies were available, were solicited. Χ2 tests and multivariable logistic regressions were conducted to evaluate the effect of the AO Region, surgeon specialisation, current position, clinical experience, and workplace on the familiarity with disease-specific PROMs, the familiarity with generic PROMs and the current use of PROMs. Exploratory factor analysis was used to identify issues underlying the extent of PROM usage.
Results 1212 surgeons completed the survey (response rate: 6.8%; margin of error: ±2.72%): 54.2% were trauma/orthopaedic surgeons, 16.6% were spine surgeons, 27.9% were craniomaxillofacial surgeons and 16 had no defined specialty. Working in a certain AO Region, surgical specialisation and current workplace were associated with a higher familiarity of disease-specific PROMs and the use of PROMs in daily clinical routine (p≤0.05). Exploratory factor analysis identified four categories important for the use of PROMs and two categories preventing the use of PROMs. In case of the availability of an adequate tool, 66.2% of surgeons would implement PROMs in clinical routine.
Conclusions Our survey results provide an understanding of the use of PROMs in clinical routine. There is consensus on the usefulness of PROMs. User-friendly and efficient tools/technologies would be a prerequisite for the daily use of PROMs. Additionally, educational efforts and/or policies might help.
- public health
- adult orthopaedics
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The sample of participating orthopaedic, trauma, spine and craniomaxillofacial surgeons could represent the general situation regarding the use of PROMs in clinical routine, with a margin of error of 2.72%.
The large number of participants (1212 surgeons) is a strength of this study. Surgeons’ answers provide an understanding of the familiarity with and the use of PROMs in clinical routine.
Reasons for the use of PROMs among surgeons already using PROMs and concerns against PROMs among surgeons not using PROMs were investigated separately; explorative factor analysis was applied.
Limitations are the low response rate of 6.8%, and a risk of selection bias associated with it.
It was assumed that surgeons interested in the topic responded to the survey. Therefore, the familiarity and current use of PROMs may have been rather overestimated.
The collection of patient-reported outcome measures (PROM) is increasingly demanded in many healthcare systems for reasons of quality control, benchmarking and reimbursement.1 It is proposed that the routine use of PROMs in clinical care could have the potential to help reform healthcare, its structure and its delivery. Healthcare providers could compare the healthcare provided with patient outcomes achieved in any patient population and use the gathered information to adjust their systems. Clinicians could use it for making treatment decisions for their patients.2
Individual physicians and/or hospitals increasingly use PROMs, however, in only a few countries, including England, Sweden and partially the USA PROMs are collected on a regional or national level.2 In England, a limited mandatory collection of PROMs was initiated by the government in 2009.3 4 In Sweden, the initiative to collect disease-specific PROMs nationwide was clinician driven.2
In 2002, the National Institutes of Health (NIH) in the USA defined the need to develop state-of-the-art PROMs as a high priority scientific project to enhance clinical research in the 21st century.5 Starting in 2004, the NIH developed the Patient-Reported Outcomes Measurement Information System (PROMIS) together with an online assessment centre1 6 to provide generic measures of the most important health outcome domains for many diseases. This freely available system enables improved efficiency and practicability for implementing PROMs in the clinical setting, using computer adaptive testing (CAT).1 6–10
A few investigations on clinicians’ attitudes to using PROMs in daily routine and/or on implications of using PROMs have been published, including surveys among paediatricians, oncologists, psychiatrists, spine surgeons and systematic literature reviews.11–19 Little information on the familiarity of trauma and orthopaedic surgeons with PROMs is available. Also, there is limited information on the use of PROMs in daily routine. This study aimed to gain detailed information on the perceived advantages and disadvantages of PROM implementation, and the important constraints and reservations for a successful implementation of such procedures.
Information was gathered from members of the AO Foundation, a medically guided non-profit organisation led by an international group of surgeons specialising in the treatment of trauma and disorders of the musculoskeletal system (www.aofoundation.org). A questionnaire was sent to 17 931 surgeons (9567 trauma, 6054 spine and 2310 craniomaxillofacial (CMF) surgeons) via SurveyMonkey, together with an invitation email explaining the purpose of the survey and a secure web link. Participation in the survey was anonymous and voluntary, and no remuneration or reward was offered for participation. No reminders were sent. As the questionnaire was developed, based on similar published surveys that evaluated surgeons’ opinions on PROMs among other medical specialties,11 17 20 no pilot survey was conducted.
Local Institutional Review Board approval has not been obtained as no medical information was collected from the participants, and data were collected and analysed anonymously. However, all participants were informed that the survey results would be published.
The questionnaire is available in online supplementary file 1. It comprised seven general questions, eliciting demographic information, the region of residence (AO Regions Africa, Asia-Pacific, Europe, Latin America, Middle East and North America), surgical specialisation, current working position, clinical experience and current workplace. Another three questions solicited surgeons’ familiarity with generic or disease-specific PROMs, and surgeons’ current use of PROMs. Detailed statements either about surgeons’ frequent use of PROMs in clinical routine, or not using PROMs in clinical routine were to be made. Therefore, if surgeons responded to currently use PROMs regularly or infrequently in daily clinical work, or both, in daily clinical work and research, their opinion on 10 detailed statements (Question A, online supplementary file 1) regarding their familiarity and usage of generic and disease-specific PROMs in clinical routine was solicited. Whereas if surgeons responded to currently not use PROMs, or to just use them for research, their opinion on seven detailed statements regarding their reasons to not use PROMs was solicited (Question B, online supplementary file 1).
In addition, information on three further aspects of PROMs was collected: the reasons why PROMs are not used more often in clinical routine (four items); what would be the most important prerequisites to implement PROMs in clinical routine (five items); and whether surgeons would use PROMs in their clinical routine (assuming the availability of tools/technologies to overcome reported obstacles) (one item).
All questions were optional, and surgeons had the possibility to skip questions.
Patient and public involvement
There was no patient involvement in the presented survey. The participating surgeons were not involved in the development of this questionnaire. The survey results will be published and made available in this way. Once the survey results are published AOTrauma, AOSpine and AOCMF will be informed so that the results can be distributed among their members.
The response rate (including completed and partially completed questionnaires) and the margin of error at 95% confidence (expressing the amount of random sampling error) were computed. Percentages for all categorical variables (excluding missing responses) were computed. To examine associations between two categorical variables, Χ2 tests were applied (univariable analyses). Multivariable logistic regression analyses were conducted to evaluate the effect of ‘region of residence’, ‘surgical specialization’, ‘current position’, ‘clinical experience’ and ‘workplace’ on (A) the familiarity with existing generic PROMs, (B) the familiarity with existing disease-specific PROMs, and (C) the current use of PROMs in clinical routine. Listwise deletion applied if a respondent stated ‘other’ for surgical specialisation, current position or workplace. The significance level was set at p<0.05.
Exploratory factor analyses (EFA) were performed for the questions relating to ‘why to use PROMs’ and for the questions ‘why to not use PROMs’ to identify broader issues underlying decisions.21 Factors were extracted using the principal component method and those factors with eigenvalues greater than 1 (Kaiser criterion) were retained.22 EFA with varimax rotation was used as a data reduction technique to create interpretable results from the retained factors.23
All statistical analyses were performed using the software SAS V.9.4 (SAS Institute).
One thousand two hundred and twelve surgeons of the AO Foundation network (17 931 surgeons; response rate: 6.8%) completed the online survey. The computed margin of error for the results of the survey was ±2.72%. Six hundred fifty-seven (54.2%) were trauma and orthopaedic surgeons, 201 (16.6%) were spine surgeons, 338 (27.9%) were CMF surgeons and 16 (1.3%) had no defined clinical specialty. Demographic data and details about the AO Region, surgical specialisation, current position, clinical experience and workplace are listed in table 1.
Overall, 661 (54.5%) reported being familiar with respective disease-specific PROMs, and 547 surgeons (45.1%) reported being familiar with generic PROMs. The univariable analysis showed that surgeons in Europe and North America were more familiar with specific PROMs (Europe: 68.5%; North America: 72.7%) and generic PROMs (Europe: 60.4%; North America: 66.4%) than those in the remaining regions (p<0.001). Spine surgeons were most familiar with both, specific PROMs (85.6%) and generic PROMs (63.7%), whereas CMF surgeons were least familiar with PROMs in general (specific: 13.6%; generic: 31.7%). ‘Surgeons’ current position’, ‘clinical experience’ and ‘workplace’ were also associated with their familiarity of PROMs (p<0.05) (table 2).
The multivariable analysis confirmed the regional differences, differences between surgeon specialties and workplaces in the familiarity with disease-specific PROMs (table 3). Of note, surgeons working either in a private or a public hospital were approximately half as likely to be familiar with disease-specific PROMs than surgeons working in a university hospital (p≤0.05). No evidence for an association between familiarity with disease-specific PROMs and ‘current position’ or ‘clinical experience’ was found (table 3).
For the familiarity with generic PROMs, above results were also confirmed in the multivariable analysis. In addition, the respondents’ ‘current position’ was found to be a significant variable in this model. Compared with registrars, consultants and senior managers or professors were more familiar with generic PROMs (p<0.05) (table 3).
Only 97/1210 (8.0%) respondents used PROMs regularly in their clinical routine, 134 (11.1%) irregularly, 94 (7.8%) for both clinical routine and research purposes, and 253 (20.9%) for research purposes only, compared with 632 (52.2%) who did not use PROMs at all. As with ‘familiarity’, the univariable analysis showed a distinct difference in the clinical use of PROMs between the different regions and specialisations: PROMs were most frequently used in Europe (32.5% of surgeons used PROMs in clinical routine or clinical research) and North America (31.3%). Spine surgeons used PROMs the most (45.3%), whereas CMF surgeons used PROMs the least (15.4%). Moreover, ‘surgeons’ clinical experience’ seemed to be associated with the routine use of PROMs (p=0.024) (table 2).
The multivariable analysis again confirmed the results from the univariable analysis, regarding ‘AO Region’, and ‘surgeons’ specialization’. Contrary to the univariable analysis, ‘workplace’ instead of ‘surgeons’ clinical experience’ was associated with use of PROMs in daily clinical routine (table 3).
Of 325 surgeons who responded to currently use PROMs in clinical routine (irregularly or regularly) or both in daily clinical work and research, 292 provided their opinions to the 10 detailed statements regarding their familiarity and routine use of PROMs. Five of the 10 statements got more than 80% agreement (either strongly agreed or agreed), and four between 67% and 78% agreement. Only 34% of surgeons agreed that they were required to document quality of life (QoL) data by the government (last statement) (table 4).
Available (non-missing) data/opinions of 281 surgeons who used PROMs in their daily routine, related to the 10 provided statements, were factor analysed. Four factors were identified with eigenvalues greater than 1 (figure 1) and accounted for 66% of the total variance. We labelled the factors as follows: factor 1: quality of care, factor 2: measurement of outcomes, factor 3: regulations and factor 4: limitations of PROMs. The main factor loadings on the 10 items are shown in figure 2.
Of 885 surgeons who responded to currently not use PROMs or to only use PROMs in research, 768 provided their opinions to the seven detailed statements regarding their reasons to not use PROMs. Four of the seven statements got less than 31% agreement (either strongly agreed or agreed), one got 47% (PROMs are too time consuming) and two statements got more than 60% agreement (60%: I am lacking sufficient information, knowledge or experience to use PROMs; 70.6%: I am interested but did not yet have the possibility to use PROMs) (table 4). For the second factor analysis, all answers of 739 surgeons to the seven items were used: two factors were identified accounting for 51% of the cumulative variance (figure 3 and 4): factor 1: overall assessment of PROMs; factor 2: institutional responses regarding to PROMs.
When surgeons were asked for further aspects considered important for an implementation of PROM-collecting instruments into the daily clinical routine, user-friendliness was very important or important for 95.2% of surgeons (938/986). But also, other aspects such as costs, time efficiency, comparability with existing software and the interpretability and clinical relevance of the results were considered important or very important (table 5).
When surgeons were asked whether they would implement PROMs into their daily routine, if tools/technologies to overcome barriers were available, 656/1212 (66.2%) agreed, whereas 335 (33.8%) were not sure yet. Of those 656 surgeons who agreed, 364 (75.4%) were using PROMs in any form at the time of the survey, and 292 (57.6%) were not using PROMs at the time of the survey (p<0.001). In a multivariable logistic regression model, surgeons from Africa were 2.67 (95% CI 1.05 to 6.78) times more likely to be willing to use a tool that would overcome barriers in collecting PROMs in clinical routine than surgeons from Europe; orthopaedic surgeons (OR: 2.11; 95% CI 1.44 to 3.09), trauma surgeons (OR: 1.90; 95% CI 1.30 to 2.79) and spine surgeons (OR: 1.74; 95% CI 1.14 to 2.66) were more likely than CMF surgeons. Surgeons working in a private hospital were half as likely (OR: 0.49; 95% CI 0.35 to 0.69) as surgeons working in a university hospital (p<0.05 for all).
Our worldwide study surveyed the familiarity of trauma/orthopaedic, spine and CMF surgeons with PROMs, their current use in clinical routine, reasons for using or not using them and respondents’ perceptions of the important aspects to be considered in implementing PROMs in clinical routine. In a multivariable regression model, familiarity with existing generic and disease-specific PROMs was influenced by ‘geographic region’, ‘surgical specialization’, ‘current workplace’ and for generic PROMs, by ‘current working position’. The use of PROMs in clinical routine was again influenced by ‘geographic region’, ‘surgical specialization’ and ‘current workplace’.
In a survey published by the Dutch Pediatric Association, 69% of 303 paediatricians reported being familiar with patient-reported QoL data.11 Another survey among the Latin-American AOSpine members reported familiarity with generic PROMs in 79% of 199 participants and familiarity with disease-specific PROMs in 86%.18 In our study, 45% of surgeons were familiar with generic and 55% with disease-specific PROMs. Interestingly, familiarity varied according to surgical specialisation and geographic region and was in general higher for disease-specific PROMs than for generic PROMs. A few European countries, that is, the UK and Sweden,3 24–26 and the USA (http://www.hosonline.org) have started collecting PROMs on a routine basis, which could be a reason for the higher local general awareness. The low overall familiarity of CMF surgeons with PROMs in our study can probably be explained by the lack of well-established PROMs for these patients. In contrast, well-established PROMs for patients with spinal disorders such as the Oswestry Disability Index27 and the Neck Disability Index28 may have contributed to the higher familiarity (of 86% of spine surgeons in our study) with PROMs within this specialty. The study by the Dutch Pediatric Association found that paediatricians working at a university hospital were significantly more familiar with the existence of QoL questionnaires than those working in a community hospital (p=0.007), and paediatric registrars were significantly less familiar with the existence of QoL questionnaires than experienced paediatricians (p=0.019).11 These results were confirmed for the surgeons/respondents participating in our survey (p<0.05).
PROMs are rather new in the field of trauma surgery and, as such, clinicians’ awareness might well be expected to be lower than in other medical disciplines. This may explain why only 26.9% of surgeons in our study used PROMs in their daily routine and only 8.0% used PROMs regularly. Higher proportions are reported from surveys which involved practitioners dealing with patients with chronic illness.11 17 20 Interestingly, ‘current working position’ and ‘level of clinical experience’ did not have a significant influence on the use of PROMs. We speculate that less experienced clinicians are not familiar with PROMs but are obliged to collect them on behalf of a more senior clinician as first point of contact.
We found a high degree of agreement among surgeons who routinely use PROMs regarding the reasons for doing so, such as prioritisation of a clinical problem, facilitation of communication between the clinician and the patient, monitoring changes/responses to the treatment, and so on. These reasons are cited in the literature and indicate that with the help of PROMs patient care and treatment can be focused more on the individual patient rather than on the disease. Moreover, the quality of healthcare delivery can be monitored and improved.18 25 28 29 Our factor analysis of the questionnaire showed that surgeons see four broad categories of reasons for using PROMs: to improve the quality of care; to contribute to the measurement of outcomes; compliance with regulation and that PROMs should be limited to the assessment of individual patient outcomes.
For policymakers, governments, regulatory bodies and funders, awareness of these four categories is important to develop incentives to increase surgeons’ motivation for using PROMs in clinical routine. At the same time, it is as important to realise why PROMs are not used more often in clinical routine. Concerns mentioned by surgeons are: absence of institutional policies to guide the process and concerns about the length of PROM questionnaires, the time to get the results, the impression that PROMs are burdensome and time consuming, and the costs and infrastructural changes related to PROM implementation.2 17–19 30 Other reasons are the lack of knowledge and experience to interpret results and to use them in clinical practice, but also doubts on the compatibility of results with routinely collected clinical measurements.11 20 30–33 Furthermore, surgeons seem to be sceptical about PROMs, because they are thought to be ‘soft and subjective’.4 34
In our study, 87.0% of all surgeons who did not routinely use PROMs disagreed with the statement that PROMs are not useful in orthopaedics and traumatology, and 61.9% agreed that they do not use PROMs due to insufficient information, knowledge or experience. Another 70.6% of surgeons stated that they would be interested in using PROMs but did not yet have the opportunity to do so. The factor analysis showed two broad constraints to the implementation of PROMs: cost and institutional ignorance and inertia. Developing policies to increase information and educational activities and improved tools/technologies for PROM use could help overcome these obstacles. Important prerequisites to implement and use PROMs were found among surgeons in our study: there was a belief in the usefulness of PROMs and a willingness to use PROMs if user-friendly technologies were available.
PROMIS instruments (http://www.healthmeasures.net) are one of the most important and promising developments in terms of advancement of PROMs. By using sophisticated algorithms, item selection is automatically matched to the health level of respondents, and the number of questions is reduced to a minimum.1 5–7 35 36 Available PROMIS instruments are useful for patients in the field of orthopaedics and traumatology (eg, the physical function CAT and pain interference CAT instruments), and even superior to established legacy instruments, due to various reasons (eg, taking less time to administer).37–45
Developments to increasingly implement PROMs into clinical routine, and even into electronic health record systems are apparent. Additionally, stand-alone software to collect PROMs is becoming available for clinical routine. Clinicians immediately see the current health status of their patients and former scores and can monitor changes over time.
The current study has limitations, mainly the rather low response rate of 6.8%, which gives rise to selection/non-response bias. Comparing our demographic survey data with available internal data derived from a membership statistic of the AO Foundation from 2018, younger surgeons aged between 25 and 34 years were underrepresented in our survey (deviations within specialties and regions: 10%–20%), surgeons aged between 35 and 44, and between 45 and 54 were over-represented (deviations: 5%–11%). CMF surgeons were over-represented in all regions (deviations: 13%–34%) and balanced by spine and trauma surgeons who were under-represented (deviations: 5%–18%). The AO Asian Pacific region was under-represented among trauma and spine surgeons (deviations: 10%–12%), while Europe was over-represented (deviations: 6%–14%). The Middle East was over-represented by 5.5%, while Latin America was under-represented by 8% among CMF surgeons. The gender distribution is considered representative within regions and specialties.
In addition, response to the survey was voluntary, and it can be assumed that mainly surgeons interested in the topic responded, which in turn may mean that the current use of PROMs is rather overestimated, that is, PROMs are even less known and less used in clinical routine than reflected by the survey.
On the other hand, the margin of error was only 2.72%, implying that the random sampling error was low. Therefore, survey estimates can be considered precise and representative for the whole population of AO surgeons. Keeping in mind that the AO Foundation has nearly 18 000 members worldwide, results could reflect the general situation regarding the use of PROMs in clinical routine.
Our research survey among trauma surgeons provides an understanding of the familiarity with PROMs and their current use in clinical routine. There is consensus on the usefulness of PROMs. A majority of surgeons is willing to implement PROMs, if an adequate tool would be available. The development of user-friendly and efficient tools/technologies both for patients and clinical staff would be an important facilitator to increase the use of PROMs in clinical routine. Additionally, education and/or policies for PROM usage in clinical routine might help.
The authors thank Ursula Saiti and the AO Clinical Divisions, namely AOTrauma, AOSpine and AOCMF for their support in distributing the online survey among their members.
Contributors AJ: conception and design of the study/survey, data collection and interpretation, manuscript drafting and revision, approval of final manuscript. CK and VK: data analysis and interpretation, revision and approval of final manuscript. AB: data interpretation, manuscript drafting and revision, approval of final manuscript. GG: data interpretation, manuscript revision and final approval.
Funding This work was financially supported by the AO Foundation via AOTrauma and AO Clinical Investigation and Documentation (AOCID).
Competing interests We received financial support from AOCID and AOTrauma for the submitted work; AJ, CK, AB and VK are employees of AOCID; no other relationships or activities that could appear to have influenced the submitted work.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement If additional data are of interest for the reader, please contact the corresponding author.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.