Article Text
PDF
Abstract
Objective To evaluate the efficacy of eLearning in the widespread standardised teaching, distribution and implementation of the Chelsea Critical Care Physical Assessment (CPAx) tool—a validated tool to assess physical function in critically ill patients.
Design Prospective educational study. An eLearning module was developed through a conceptual framework, using the four-stage technique for skills teaching to teach clinicians how to use the CPAx. Example and test video case studies of CPAx assessments were embedded within the module. The CPAx scores for the test case studies and demographic data were recorded in a secure area of the website. Data were analysed for inter-rater reliability using intraclass correlation coefficients (ICCs) to see if an eLearning educational package facilitated consistent use of the tool. A utility and content validity questionnaire was distributed after 1 year to eLearning module registrants (n=971). This was to evaluate uptake of the CPAx in clinical practice and content validity of the CPAx from the perspective of clinical users.
Setting The module was distributed for use via professional forums (n=2) and direct contacts (n=95).
Participants Critical care clinicians.
Primary outcome measure ICC of the test case studies.
Results Between July and October 2014, 421 candidates from 15 countries registered for the eLearning module. The ICC for case one was 0.996 (95% CI 0.990 to 0.999; n=207). The ICC for case two was 0.988 (0.996 to 1.000; n=184). The CPAx has a strong total scale content validity index (s-CVI) of 0.94 and is well used.
Conclusions eLearning is a useful and reliable way of teaching psychomotor skills, such as the CPAx. The CPAx is a well-used measure with high content validity rated by clinicians.
- EDUCATION & TRAINING (see Medical Education & Training)
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
A number of new functional assessment tools have been developed and validated to assess and measure intensive care unit-acquired weakness (ICUAW); however, no educational package to allow widespread and standardised implantation of these tools in clinical practice exists.
This study demonstrated that eLearning modules aimed at teaching clinicians how to use functional assessment tools, such as the Chelsea Critical Care Physical Assessment (CPAx) tool, can be an effective way of delivering worldwide-standardised education and facilitate consistent clinical implementation.
This study supports the inter-rater reliability of the CPAx tool; however, as these data are based on video footage of patient assessments, not real-time patients, there may be some confounders, such as environmental and proprioceptive factors that limit the extrapolation to clinical practice.
The CPAx is a widely used measure in England (43.3% of adult ICUs), and its use has facilitated implementation of the National Institute for Health and Care Excellence (NICE) guidelines for Rehabilitation after Critical Illness (March 2009).
Further research is required to explore how functional assessment tools such as the CPAx can be used to facilitate the rehabilitation process and how eLearning can be used in the teaching of psychomotor tasks.
Introduction
Intensive care unit-acquired weakness (ICUAW) often leads to prolonged periods of mechanical ventilation and life-changing disability,1 with patients demonstrating reduced fitness and health-related quality of life up to 5 years after discharge.2 For these patients, early rehabilitation and mobilisation is essential, facilitating liberation from mechanical ventilation, optimising recovery and improving the chance of survival.3–5 As more people are surviving critical illness, the problem of ICUAW and the health economic impact is growing. Hence, in 2009, the National Institute for Health and Care Excellence (NICE) introduced clinical guideline 83 (CG83), entitled ‘Rehabilitation after Critical Illness’.3 This guideline recommends regular comprehensive assessment of physical disability associated with ICUAW, but due to a dearth of validated measurement tools, they were unable to make recommendations regarding the most appropriate objective assessment tool to use.
A number of measurement systems have been developed in the past 5 years to address this and allow the accurate objective assessment and monitoring of ICUAW. These include the Chelsea Critical Care Physical Assessment (CPAx) tool,6–8 the Physical Function ICU Test-scored (PFIT-s)9 ,10 and the Functional Status Scale for ICU (FSS-ICU).11 As ICUAW is a latent trait, under classical test theory, all of these measures require psychometric evaluation for reliability, validity and responsiveness. This has been completed to varying degrees with each of these scores and all have demonstrated strengths and limitations.12
The CPAx tool specifically has demonstrated validity and reliability in the general ICU population and responsiveness in the burn ICU population.6–8 In the development of the CPAx, one of the key aims was simplicity around teaching, implementation, utility and understanding. Traditionally, physical therapy assessment tools are distributed through the use of scientific journals. Most of these tools have no training resource or come with simple instructions. Some of the more complex systems, for example, the Functional Assessment Measure (FAM), mandate that therapists attend a training course; this has obvious implications in terms of resources.13 Hence, in-house teaching from senior to junior clinicians has become the norm. This informal teaching is important in the assessment of a psychomotor task; however, a more structured consistent teaching method would be beneficial to ensure reliable tool administration. One such approach is via an eLearning package.
eLearning allows students to access educational resources globally, which are repeatable, cheap and convenient (provided they have the internet access and hardware required).14 ,15 They can also be interactive, with video demonstrations that help to facilitate the learning of practical skills. Hence, this may be a novel and effective way of distributing practical education on the consistent use of assessment tools such as the CPAx.
The aims of this project were to:
Analyse the use of eLearning as a method for teaching clinicians how to implement the CPAx tool consistently.
Evaluate the content validity of the CPAx tool from the perspective of those currently using it.
Method
Ethical approval
Under current UK research governance rules, this project was defined as service improvement and did not require research ethics approval or formal consent of participants.
Content and development of the eLearning module
To establish a conceptual framework and learning objectives for the CPAx eLearning module, meetings with senior physiotherapists using the CPAx tool within London were conducted. This highlighted the essential content of the eLearning module. Applying the four-stage technique of skill acquisition (figure 1)16 to this information, the following learning objectives and module chapters were identified:
Four-stage technique of skill acquisition.16
Learning objectives
Primary
To understand how to use and implement the CPAx in clinical practice.
Secondary
To understand the rationale for the development of the CPAx tool.
To understand the CPAx development process.
eLearning module chapters
Registration page to collect demographics (ie, job role, contact details, workplace and the number of years of clinical experience).
An explanation of the development of the CPAx.
Breakdown of how the CPAx tool is administered.
Video footage of a demonstration CPAx assessment and discussion between the two physiotherapists completing the assessments to provide the rationale for each score given.
Two interactive test case studies for the learner to assess on the CPAx are included. Following completion of their assessment, an ideal score is revealed and a video of discussion between two physiotherapists to explain why that score was allocated.
Recommendations on how to use and implement the CPAx.
Once the content of the module was compiled, it was formatted into PowerPoint and sent out for review by the senior physiotherapists involved. The final edit was formatted into the online eLearning module, designed to take 40–60 min to complete. The web link to the module is http://cpax.ocbmedia.com.
The data from the registration page and the CPAx scores for the test case studies are stored in a secure area in the website. Once the module is complete, a certificate is provided and the CPAx documentation is available for download in pdf format.
Case study participants
For the video case studies, three current patients with ICUAW were approached. Patients were intentionally selected to represent three different scores on the CPAx scale. This is because the CPAx is an ordinal scale, so the consistency of scoring needed to be evaluated throughout.
Permission to film patients was granted by the Caldicott Guardian and Communications manager at the local hospital. A patient information sheet was provided and written informed consent was gained. As this was considered an educational tool, the need for formal approval was waived. The consent process was video recorded.
The video case studies were a real-time video recording of the patient being assessed by their familiar, treating physiotherapist(s) either in their usual place of therapy, that is, the ICU bed space, or the gym. There was no change to the clinical care of the patient during this time. The lead researcher and cameraman directed the camera angles to try to ensure that all components of the CPAx assessment were captured. The videos were edited and embedded within the eLearning module.
Pilot
The eLearning module was sent out via email for pilot testing to a convenience sample of eight acute hospitals. Those data collected from the case studies were tested for reliability using intraclass correlation coefficients (ICCs). Following completion of the module, an electronic survey to analyse the usability of the eLearning module was distributed to identify any improvements needed (see online supplementary appendix 1).
Supplementary appendix 1
During the pilot stages, 61 participants registered for the module, from 18 UK and 1 Greek hospital. The eLearning module is reported automatically as ‘completed’ or ‘started’ depending on whether all chapters have been accessed and finished; 26 candidates completed the entire module. A total of 30 people completed case study 1 and 28 completed case study 2; both case studies were accessible to all registrants. The ICC for average measures for pilot case study 1 (n=30) was 0.989 (95% CI 0.976 to 0.997). The ICC for pilot case study 2 (n=28) was 0.981 (95% CI 0.959 to 0.994).
Changes made following the pilot
A total of 17 (65.4%) of the 26 ‘completers’ responded to the evaluation questionnaire, which led to some changes to the content of the module: the video footage was shortened; a statement was included to advise candidates to complete the module in a quiet room with adequate audio volume; the CPAx development team members were described; a CPAx level descriptor pdf document was made available for download in the case study chapter of the module for cross-referencing with the case study. Owing to poor correlation between raters in the assessment of sitting balance, case study 2 was refilmed with a different patient at a similar functional level, to determine if this was due to reliability of the CPAx, image quality or inadequacies of psychomotor education via two-dimensional (2D) imagery.
Distribution and evaluation of the eLearning module
The final link to the eLearning module was distributed for use on 14 July 2014 via the Chartered Society of Physiotherapy (CSP) forum, on the MedConcert ICU Recovery Network, and emailed out to all interested centres (n=95). On 9 October 2014, the two test case study CPAx scores were extracted for analysis. One year after distribution of the module, an electronic CPAx utility questionnaire was emailed to all eLearning module registrants (see online supplementary appendix 2). The questionnaire also contained two content validity questions (questions 11 and 12) asking clinicians to rate the content of the CPAx, that is, the relevance of the component parts and the appropriateness of the breakdown of the level descriptors. These data were used to determine the content validity of the CPAx tool using an item and a total scale content validity index (i-CVI and s-CVI, respectively)17; this is an index of the proportion of clinicians endorsing the content of the tool. Expert content validity of the CPAx had already been established in the initial development of the tool.4 The purpose of this further testing was to establish the CVI of the CPAx in those physiotherapists using the tool regularly in clinical practice.
Supplementary appendix 2
Statistical analysis
Data were analysed using Prism 6 (GraphPad, La Jolla, California, USA) and IBM SPSS Statistics V.22. The total CPAx score and individual breakdown of the scores were tested for normality using D'Agostino and Pearson omnibus normality test; where normally distributed, scores were reported as means and SDs; where data were skewed, median range and IQR were reported. ICC for agreement between raters for a two-way random model for absolute agreement was analysed.
The CPAx utility questionnaire results are presented using descriptive statistics, primarily percentages. The i-CVI is the number of clinicians rating the components (items) of the CPAx and the Guttman scaling as ‘highly’ or ‘quite’ relevant divided by the total number of clinicians. The s-CVI is the mean of the individual i-CVI's for each component part. A figure of above 0.80 is considered acceptable content validity.17
Results
Demographics
Table 1 shows the demographics of the healthcare professionals registered for the eLearning module.
Demographics of module registrants between July and October 2014
Intraclass correlation coefficient (reliability)
The median (range and IQR) for each component part of the CPAx is reported in table 2.
Median (range and IQR) of case study CPAx scores
The ICC for case one (n=207 items) was 0.996 (95% CI 0.990 to 0.999). The ICC for case two (n=186 items) was 0.988 (0.996 to 1.000). When divided into quartiles by years of experience, there was no difference in the ICC for each subgroup.
Content validity index
Table 3 shows the i-CVI and s-CVI for the CPAx score.
Content validity index
CPAx utility questionnaire
On 4 August 2015, 971 clinicians had accessed the eLearning module, all of which were emailed the utility questionnaire. A total of 67 of these emails did not go through, leaving a final total of 899 candidates.
CPAx usage
The total questionnaire response rate was 37.4% (n=336/899); however, some participants did not respond to some of the questions. As a result, the sum of the response rate for the individual questions does not always equal the sum of the response rate for the survey.
Of the respondents, 76.7% (n=262/333 respondents) used the CPAx in clinical practice. The majority of these were in the UK (n=226). In England, there was at least one clinician from 43.3% of all adult acute National Health Service (NHS) trusts (n=68/157); further users were in Ireland, Switzerland, the USA, Australia, South Africa and Canada.
Of those reporting to use the CPAx (n=262), 56.5% (n=130) of question responders (n=230) did not use any other assessment tool; 14.4% (n=33/230) had a locally designed assessment tool; 16.5% (n=38) used the Medical Research Council (MRC) sumscore of muscle strength and 15.7% (n=36) used grip strength dynamometry.
Of those not using the CPAx (n=71), 45.8% (n=27) of question responders (n=59) were not using any formal functional assessment tool and 27.1% (n=16) had a locally designed assessment form. However, a number of other measures were being used. The most common were the MRC sumscore (n=11; 18.6%), hand-grip dynamometry (n=7; 11.9%), the Barthel Index (n=6; 10.2%) and the Functional Independence Measure (n=6; 10.2%).
The primary reasons reported by question respondents (n=211) for using the CPAx were that it was simple to use (n=167; 79.2%), validated in this population (n=148; 70.1%), quick to complete (n=110; 52.1%) and easy for staff to understand (n=130; 61.6%); 13.7% (n=29) of clinicians reported that patients found this the most useful measure, and 8.5% (n=18) said that it was recognised by other services.
Question respondents (n=261) reported using the CPAx to assess patients three times a week (n=64; 24.5%), twice a week (n=48; 18.4%), once per week (n=59; 22.6%) or when the patient showed signs of improvement (n=25; 9.6%). The CPAx was used routinely on all critical care admissions by 33.5% (n=87) of question responders (n=260); however, the majority had inclusion criteria: ICU length of stay of over 48 h (n=44; 25%) or over 72 h (n=46; 26.1%); a set period of time on mechanical ventilation (n=11; 19.4%); diagnosed ICUAW (n=46; 26.1%); suspected ICUAW (n=62; 35.2%); patients categorised as level 2, that is, high dependency (n=21; 11.9%), or level 3, that is, intensive care patients (n=21; 11.9%) or the presence of a tracheostomy (n=24; 13.64%).
CPAx ‘pros’ and ‘cons’
Two (0.66%) of all respondents (n=305) reported no benefits in using the CPAx; 83.9% (n=256) felt that it helped the clinician to monitor patient progress; 59.0% (n=180) felt that it helped the patient to monitor their own progress; 48.5% (n=148) felt that it gave structure to the physical assessment; 32.1% (n=98) felt that it helped to teach junior staff; 35.4% (n=108) found that it improved team communication; 58.7% (n=179) found that it assisted in adherence with NICE CG833; 72.1% (n=220) felt that it assisted in therapy goal setting and 41.0% (n=125) found that it helped to motivate patients during rehabilitation.
When asked about the negatives of the CPAx, 53.9% (n=138) of question respondents (n=256) did not report any negatives. However, 16.0% (n=41) reported that it was too time-consuming; 5.1% (n=13) reported that it was too complex; 5.1% (n=13) felt that it was a ‘paper exercise’; 12.5% (n=32) said that it can be demotivating for the patients if they do not make progress; 7.4% (n=19) said it was not responsive enough and 6.6% (n=17) found some discrepancies between raters. The hand-grip component was also felt to be complex by 5.0% (n=13) of completers.
Clinicians' perception of the eLearning module
The majority (90.3%; n=250) of question respondents (n=279) felt that following completion of the eLearning module, they understood how to use the CPAx tool. The module was considered simple to use by 93.6% (n=264) of respondents; 77.3% (n=214) felt that they could use the CPAx consistently after completing the module; however, 21.3% (n=59) felt that further in-house teaching is required and 91.0% of candidates felt competent to explain how the CPAx score worked to colleagues following completion of module.
Discussion
Clinical significance of findings
This is the first study to develop and evaluate the use of eLearning to educate clinicians on the use of a critical care functional assessment tool. The use of eLearning allowed widespread and rapid distribution of the CPAx, accessed by clinicians globally. It also allowed evaluation of the consistency of scoring following education via eLearning, evaluation of the clinical uptake of the tool and clinician content validity of the CPAx.
The between-rater ICC for CPAx scores was strong and consistent between case studies, independent of clinician experience. The ICC results are also consistent with the previously published inter-rater reliability (IRR) data on the CPAx, when patients' were assessed in real time following face-to-face practical teaching.4 This supports the use of eLearning as a tool for educating psychomotor tasks in physiotherapy.
The ICC data may also support the IRR of the CPAx tool in clinical practice. Assessment of IRR for critical care physical assessment tools is difficult due to the nature of critical illness: if consecutive assessments are completed in 1 day, patient fatigue may influence the results; and if patients are assessed on consecutive days, the clinical presentation may have changed due to the labile nature of critical illness or medical interventions, for example. If observed assessments or video recordings are used, the physiotherapist does not get the proprioceptive feedback, but they do get a consistent clinical presentation to compare. Hence, all methods are problematic and it is the authors' views that these data could be extrapolated to support the IRR of the CPAx tool itself.
Clinicians using the CPAx tool rated the content validity as high (s-CVI 0.94). This corresponded with the experts’ (n=14) CVI rating in the development stages (2-CVI 1.00).4 The high content validity is reflected in the clinical uptake of the measure (76.7%). Most respondents found the CPAx simple; useful for staff education, clinical assessment and goal setting; and responsive to change. It also improved clinical compliance with NICE CG83. Furthermore, a recent systematic review of measurement instruments to assess ‘impairment and physical limitation’ in the critically ill patient12 concluded that the PFIT-s and the CPAx are the most thoroughly scrutinised robust measures. Combined, these findings suggest that the CPAx is a useful, valid measure that can be implemented with relative ease and could improve the quality of clinical care.
Methodological limitations
Although the ICC data are positive, the scoring consistency of ‘dynamic sitting’ in case study 1 and ‘standing balance’ in case study 2 had a two-point variation in the IQR. This could be due to the breakdown of the CPAx tool level descriptors or the use of eLearning as a method of teaching. The wording in the level descriptors varies in objectivity, for example, a hoist transfer from bed to chair has limited subjectivity, whereas the definition of ‘minimal’ versus ‘moderate’ assistance to maintain ‘dynamic sitting’ or ‘standing balance’ is more open to interpretation. This may reduce the reliability of certain sections of the CPAx, but it is a common and unavoidable issue in scoring systems of this kind. Also, the use of a 2D image in the context of a psychomotor action that relies on proprioceptive feedback for the assessor—specifically when differentiating between the amounts of physical assistance required for a task—may have limits.
In the pilot stages, inconsistency between raters in the sitting balance component of the CPAx score was noted, and the pilot questionnaire feedback showed that the amount of assistance required to maintain sitting balance was difficult to interpret from the video footage. This case study was refilmed with a different patient to optimise the image quality and determine if the poor correlation between raters was an issue with the intra-rater reliability of the CPAx tool, or due to inadequacies of psychomotor education via 2D eLearning.
The response rate to the CPAx utility questionnaire was 37.4% (n=336/899). It is possible that those with strong opinions on the CPAx tool were more likely to reply, representing a response bias.
Other factors that may influence the response rate include one candidate responding on behalf of a hospital team, and rotational staff moving into different clinical specialties. Furthermore, the primary developer of the CPAx distributed the utility questionnaire potentially influencing the responses, however the questionnaire could be responded to anonymously mitigating this risk.
Future research recommendations
Further research into the use of eLearning as a method of teaching psychomotor tasks in this clinical specialty would be beneficial to explore the long-term learning effect and cost-benefit analysis.
A small proportion of candidates felt that the CPAx was time-consuming, complex to administer and that lack of change could be demotivating for patients. The grip strength component of the CPAx was reported as the least relevant component, with only 72% endorsing its inclusion. The practicalities of how to implement the CPAx efficiently, local factors such as CPAx data recording, and how it is communicated with patients, exploring the impact on goal setting and motivation, warrant evaluation.
As the CPAx is now being used in non-English-speaking countries with different physiotherapy practices, validation of translation is essential, to address language issues and local contextual differences. It may be appropriate to consider removal of the grip strength component of the CPAx; however, this would necessitate further in-depth validation of the modified CPAx tool.
Conclusion
eLearning is an effective mode of delivering education in a large geographical area on the consistent and reliable use of the CPAx functional assessment tool. This has facilitated compliance with NICE CG83. The CPAx is endorsed by clinical users of the tool. In addition, eLearning modules may have utility as evaluation tools.
Acknowledgments
The authors acknowledge the contribution of the physiotherapists and clinicians involved in the initial development of the CPAx tool. They also acknowledge the support of OCB Media Ltd. and Directions Films Ltd. staff and patients at Chelsea and Westminster NHS Foundation Trust and Barts Health NHS Foundation Trust for assistance in the filming of the case studies.
References
Footnotes
Contributors EJC, JMH and SJB were all involved in the conception of the study design, data collection and analysis and the content and write-up of this manuscript.
Funding Funding for time allocated to this project was received from Chelsea and Westminster Health Charity in conjunction with Westminster Medical School Research Trust (charity number 247808). The research was further supported by the National Institute for Health Research (NIHR) Biomedical Research Centre based at Imperial College Healthcare NHS Trust and Imperial College, London. The views expressed are those of the authors and not necessarily of The Department of Health, NHS or the NIHR.
Competing interests None declared.
Ethics approval Chelsea and Westminster NHS Foundation Trust Research and Development team and Caldicott Guardian.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.