Article Text
PDF
Abstract
Introduction Insufficient skills in drug dose calculations increase the risk for medication errors. Even experienced nurses may struggle with such calculations. Learning flexibility and cost considerations make e-learning interesting as an alternative to classroom teaching. This study compared the learning outcome and risk of error after a course in drug dose calculations for nurses with the two methods.
Methods In a randomised controlled open study, nurses from hospitals and primary healthcare were randomised to either e-learning or classroom teaching. Before and after a 2-day course, the nurses underwent a multiple choice test in drug dose calculations: 14 tasks with four alternative answers (score 0–14), and a statement regarding the certainty of each answer (score 0–3). High risk of error was being certain that incorrect answer was correct. The results are given as the mean (SD).
Results 16 men and 167 women participated in the study, aged 42.0 (9.5) years with a working experience of 12.3 (9.5) years. The number of correct answers after e-learning was 11.6 (2.0) and after classroom teaching 11.9 (2.0) (p=0.18, NS); improvement were 0.5 (1.6) and 0.9 (2.2), respectively (p=0.07, NS). Classroom learning was significantly superior to e-learning among participants with a pretest score below 9. In support of e-learning was evaluation of specific value for the working situation. There was no difference in risk of error between groups after the course (p=0.77).
Conclusions The study showed no differences in learning outcome or risk of error between e-learning and classroom teaching in drug dose calculations. The overall learning outcome was small. Weak precourse knowledge was associated with better outcome after classroom teaching.
- HEALTH SERVICES ADMINISTRATION & MANAGEMENT
- MEDICAL EDUCATION & TRAINING
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
-
Medication errors are often considered to be due to stressful working situations, lapses in attention or disturbances. This study demonstrates that more basic and continuing training in drug dose calculations is needed to prevent errors.
-
The method includes a certainty evaluation of each drug dose calculation, and a development of a new risk of error measurement framework.
-
Choosing between classroom teaching and e-learning does not solve the underlying problem with poor numeracy.
-
Controlled test conditions may be regarded as a limitation. Although the testing of drug dose calculations is perceived as stressful in itself, it is not equal to the time pressure or disturbances in a working situation when errors occur.
Introduction
From international reviews and reports of adverse drug events, incorrect doses account for up to one-third of the events.1–3 Many health professionals find drug dose calculations difficult. The majority of medical students are unable to calculate the mass of a drug in solution correctly, and around half the doctors are unable to convert drug doses correctly from a percentage concentration or dilution to mass concentration.4 ,5 Nurses carry out practical drug management after the physicians’ prescriptions both in hospitals and primary healthcare. In Norway, a faultless test in drug dose calculations during nursing education is required to become a registered nurse.6 Both nursing students and experienced nurses have problems with drug dose calculations, and nursing students early in the programme showed limited basic skills in arithmetic.7–10 We have shown a high risk of error in conversion of units in 10% of registered nurses in an earlier study.11
E-learning was introduced with the internet in the early 1990s, and has been increasingly used in medical and healthcare education. E-learning is independent of time and place, and the training is easier to organise in the health services than classroom teaching, and at a lower cost. A meta-analysis from 2009 summarised more than 200 studies in health professions education, and concluded that e-learning is associated with large positive effects compared with no intervention, but compared with other interventions the effects are generally small.12 There is a lack of drug dose calculation studies where different didactic methods are compared.
The objective of this study was to compare the learning outcome, certainty and risk of error in drug dose calculations after courses with either self-directed e-learning or conventional classroom teaching. Further aims were to study factors associated with the learning outcome and risk of error.
Methods
Design
A randomised controlled open study with a parallel group design.
Participants
Registered nurses working in two hospitals and three municipalities in Eastern Norway were recruited to participate in the study. Inclusion criteria were nurses with at least 1 year of work experience in a 50% part-time job or more. Excluded were nurses working in outpatient clinics, those who did not administer drugs and any who did not master the Norwegian language sufficiently. The study was performed from September 2007 to April 2009.
Interventions
At inclusion, all participants completed a form with relevant background characteristics, and nine statements from the General Health Questionnaire (GHQ 30).13 Quality of Life tools are often used to explore psychological well-being. The GHQ 30 contains the dimensions of a sense of coping and self-esteem/well-being, and was used to evaluate to what extent the nurses’ sense of coping affected their calculation skills. The nurses performed a multiple choice (MCQ) test in drug dose calculations. The questions were standard calculation tasks for bachelor students in nursing at university colleges. The test was taken either on paper or on an internet website. The time available for the test was 1 h, and the participants were allowed to use a calculator.
After the test, the nurses were randomised to one of two 2-day courses in drug dose calculations. One group was assigned to a self-directed, interactive internet-based e-learning course developed at a Norwegian university college. The other was assigned to a 1-day conventional classroom course and a 1-day self-study. The content of the two courses was the same: a review of the basic theory of the different types of calculations, followed by examples and exercises. The topics covered were conversion between units; formulas for dose, quantity and strength; infusions; and dilutions. The e-learning group continued with interactive tests, hints and suggested solutions. They had access to a collection of tests with feedback on answers, and a printout of the compendium was available. The classroom group had 1 day lecture covering the basic theory; exercises in groups; discussion in a plenary session and an individual test at the end of the day. The second day was self-study, with a textbook including exercises used at the same college.14 Two to four weeks after the course, the nurses were retested in drug dose calculations with a similar MCQ test as the pretest.
Sample size
Studies testing drug dose calculations in nurses have shown a mean score of 75% (SD 15%).15–17 In a study with 14 questions, this is equivalent to a score of 10.5 (SD 2.1). To detect a difference of one correct answer between the two didactic methods with a strength of 0.8 and α<0.05, it was necessary to include 74 participants in each group. Owing to the likely dropouts, the aim was to randomise 180 participating nurses.
Randomisation
At inclusion, each nurse was stratified according to five workplaces: internal medicine, surgery or psychiatric wards in hospitals, and nursing home or ambulatory care in primary healthcare. Immediately after submission of the pretest, the participants were randomised to one of the two didactic methods by predefined computer-generated lists for each stratum.
Data collection
Participant characteristics
The following background characteristics were recorded: age; gender; childhood and education as a nurse in or outside of Norway; length of work experience as a nurse in at least a 50% part-time job; part-time job percentage in the past 12 months; present workplace in a specific hospital department (surgery, internal medicine or psychiatry) or primary healthcare (nursing home or ambulatory care); and frequency of drug dose calculation tasks at work, score 0–3: 0=less than monthly, 1=monthly, 2=weekly, 3=every working day. Further educational background was recorded (yes/no): mathematics beyond the first mandatory year at upper secondary school; other education prior to nursing; postgraduate specialisation and courses in drug dose calculations during the past 3 years. The participants registered motivation for the courses in drug dose calculations, rated as 1=very unmotivated, 2=relatively unmotivated, 3=relatively motivated, 4=very motivated.
In addition, the participants were asked to consider statements from GHQ 30, in the context of performing medication tasks: five regarding coping (finding life a struggle; being able to enjoy normal activities; feeling reasonably happy; getting scared or panicky for no good reason and being capable of making decisions), and four regarding self-esteem/well-being (overall doing things well; satisfied with the way they have carried out their task; managing to keep busy and occupied; and managing as well as most people in the same situation). The ratings of these statements were 0–3: 0=more/better than usual, 1=as usual, 2=less/worse than usual and 3=much less/worse than usual; ‘as usual’ was defined as the normal state.
Outcomes
Drug dose calculation test and certainty in calculations
A drug dose calculation test was performed before and after the course: 14 MCQs with four alternative answers. The topics were as follows (number of questions in brackets): conversion of units,7 formulas for calculation of dose, quantity or strength,4 infusions2 and dilutions.1 For each question, the participants indicated a self-estimated certainty, graded from 0 to 3: 0=very uncertain, and would search for help; 1=relatively uncertain, and would probably search for help; 2=relatively certain, and would probably not search for help; and 3=very certain, and would not search for help. The questionnaires used are enclosed as online supplementary additional file 1.
Risk of error
Risk of error was estimated by combining knowledge and certainty for each question rated on a scale from 1 to 3, devised for the study. Correct answer combined with relatively or high certainty was regarded as a low risk of error (score=1), any answer combined with relatively or very low certainty was regarded as a moderate risk of error (score=2), and being very or relatively certain that an incorrect answer was correct was regarded as a high risk of error (score=3).
Course evaluation
After the course, the nurses recorded their assessment of the level of difficulty of the course related to their own prior knowledge (1=very difficult, 2=relatively difficult, 3=relatively easy, 4=very easy); and course satisfaction (1=very unsatisfied, 2=relatively unsatisfied, 3=relatively satisfied, 4=very satisfied). An evaluation of the usefulness of the specific course in drug dose calculations in daily work as a nurse was rated from 1=very small, 2=relatively small, 3=relatively large to 4=very large.
Ethical considerations
All participants gave written informed consent. The tests were performed de-identified. A list connecting the study participant number to the names was kept until after the retest, in case any of the participants had forgotten their number. To protect the participants from any consequences because of the test, the data were made anonymous before the analysis.
Even if the study might uncover that individuals showed a high risk of medication errors due to lacking calculation skills, it was considered ethically justifiable not to be able to expose their identity to their employer.
Data analysis
The analysis was performed with intention-to-treat analyses. In addition, a per protocol analysis was performed for the main results. Depending on data distribution, comparisons between groups were analysed with a χ2 or Fisher’s exact test, a t test or Mann-Whitney U test, analysis of variance, Friedman, and Pearson or Spearman tests for correlations, and a Wilcoxon signed-rank test for paired comparisons before and after the course. All variables possibly associated with the learning outcome and change in risk of error were entered in linear regression analyses to identify independent predictors.18 Two-tailed significance tests were used, and a p value <0.05 was considered statistically significant.
The protocol contained instructions for handling missing data. Unanswered questions were scored as ‘incorrect answer’, and unanswered certainty scores as ‘very uncertain’. For participants who did not take the test after the course, the result from the pretest (last observation) was carried forward. The analysis was performed with SPSS V.18.0 (SPSS Inc, Chicago, Illinois, USA). All results are given as the mean and (SD) if not otherwise indicated.
Results
In total, 212 registered nurses were included in the study, and 183 were eligible for randomisation. Figure 1 shows the flow of participants throughout the study, and table 1 summarises the participant characteristics and the pretest results. The two groups were well balanced with respect to baseline characteristics. Of the 183 nurses, 79 (43%) were recruited from hospitals (48 from surgery departments, including intensive care units; 23 from internal medicine wards; 8 from psychiatric wards) and 104 (57%) from primary healthcare (52 from nursing homes and 52 from ambulatory healthcare). Nearly half of the nurses (48%) performed drug dose calculations weekly or more often.
Participant flow chart.
Participants’ characteristics and pretest results
There was a tendency for more dropouts in the e-learning group: 18.4% vs 9.9% (p=0.10). The dropouts did not differ from those who completed the study regarding the workplace: 12 from hospitals and 14 from primary healthcare (p=0.74), or pretest result: score 10.5 vs 11.1, 95% CI for difference −1.5:+0.2 (p=0.13).
Knowledge, learning outcome and risk of error
The test results before and after the course are shown in figure 2, and the upper part of table 2 gives the main results after e-learning and classroom teaching. No significant difference between the two didactic methods was detected for the overall test score, certainty or risk of error. The overall knowledge score improved from 11.1 (2.0) to 11.8 (2.0) (p<0.001). Before and after the course, 20 (10.9%) and 37 (20.2%) participants, respectively, completed a faultless test. The overall risk of error decreased after the course from 1.5 (0.3) to 1.4 (0.3) (p<0.001), but 41 nurses (22%) showed an increased risk, 20 from the e-learning group and 21 from the classroom group. This proportion is within the limits of what could appear by coincidence from a normal distribution (24%), and with a mean learning outcome of 0.7 (0.2).
Test results in drug dose calculations.
Main results after course in drug dose calculations
An analysis of the 141 participants who completed the study according to the protocol did not alter the main finding that there was no difference between the two didactic methods. The overall knowledge score improved from 11.1 (2.0) to 12.0 (2.0) (p<0.001).
Table 3 gives the results as the proportion of correct answers and the proportion of answers with a high risk of error within each calculation topic before and after the course. The test results in each topic for the two didactic methods showed that the classroom group scored significantly better after the course in conversion of units: 86% correct answers vs 78% (p<0.001), with no difference in the other topics. Overall, there were significant differences between the four topics in knowledge and risk of error both before and after the course, p<0.001 (Friedman's test). Sense of coping or self-esteem/well-being was not affected by the course for either of the groups, data not shown.
Knowledge and high risk of error within each calculation topic before and after course
Factors significantly associated with good learning outcome and reduction in the risk of error after the course are given in table 4. Among these factors, the randomisation to classroom teaching was significantly better in learning outcome, adjusted for other variables. Both low pretest knowledge and certainty score were associated with a reduced risk of error after the course, as were being a man and working in hospital. Self-evaluations of coping and self-esteem/well-being were neither associated with learning outcome nor with risk of error. The total R2 changes for the variables significantly associated with good learning outcome and risk of error were 0.28 and 0.18, respectively.
Factors significantly associated with learning outcome and reduction in risk of error after course in drug dose calculations
Course evaluation
Nearly all (97.5%) of the participants stated a need for training courses in drug dose calculations.
The evaluation after the course showed no difference between the didactic methods in the expressed degree of difficulty or course satisfaction, data not shown. The specific value of the course for working situations was scored 3.1 (0.7) in the e-learning group and 2.7 (0.7) in the classroom group (p<0.001).
Auxiliary analyses
A post hoc analysis for subgroups with a pretest knowledge score ≥9 and <9 is given in the lower part of table 2. For participants with a low prescore, classroom teaching gave a significantly better learning outcome and reduced risk of error after the course. The overall knowledge score improved in the high score group from 11.6 (1.4) to 12.0 (1.9) and in the low score group from 7.2 (1.0) to 9.9 (2.3), and the difference in learning outcome was highly significant (p<0.001).
Discussion
Drug dose calculation skills
The study was not able to demonstrate an overall difference in learning outcome between the two didactic methods, either of statistical or clinical importance. Both methods resulted in improvement of drug dose calculations after the course, although the learning outcome was smaller than what was defined as clinically relevant. Adjusted for other contributing factors for learning outcome in the multivariable analysis, the classroom method was statistically superior to e-learning, and so was the case for a subgroup with a low pretest result. This finding from the post hoc analysis was probably the only outcome that could have a meaningful practical implication for choice of learning strategy, if reproduced in new studies. These results were in accordance with a meta-analysis of 201 trials comparing e-learning with other methods.19 The review summarised that any educational action gives a positive outcome, regardless of the method. E-learning works compared with no intervention, but tested against conventional methods it is difficult to detect any differences.
Drug dose calculations are not advanced in a mathematical sense. The basic arithmetic functions of addition, subtraction, multiplication or division are needed to decide decimals and fractions. What seems to be challenging is to conceptually understand the difference in information from the concentration denomination: per cent or mass per unit volume, or the ability to set up the right calculation for the relationship between dose or mass, volume or amount and concentration or strength. A standard labelling to mass per unit volume has been strongly recommended.20 ,21
The fact that only 1 out of 10 nurses performed a faultless pretest was not surprising, from what is previously shown. In a study by McMullan, only 5% of the nurses achieved 80% correct calculations.22 Although statistically significant, the limited overall learning outcome after the courses was somewhat disappointing, with only 2 out of 10 with faultless tests. It seemed that the incorrect calculations were more frequent in conversion of units, the least complex task in the mathematical sense. The conversion of units improved the most after the course, while the learning outcome in the arithmetic tasks of infusions and dilutions were unchanged. This has also been observed by other investigators, and supports the view that the challenges in drug dose calculations are more likely due to a poor conceptual understanding.10 Recent papers address the importance of including conceptual (understanding the problem), calculation (dosage computation) and technical measurement (dosage measurement) competence in teaching nurses in vocational mathematics, with models to help them understand the ‘what’, the ‘why’ and the ‘how’ in dosage problem solving.23 ,24
Risk of error
The study was not able to demonstrate any difference in the risk of error between the e-learning and classroom groups, either before or after the course. Asking for certainty in each calculation made it possible for the nurses to express whether they normally would have consulted others or not when doing the calculation. Being certain that an incorrect answer was correct was regarded as an adequate estimate for a high risk of error. To the best of our knowledge, such a method for estimating a risk of error from a test situation is not described by others, and may be a contribution to future research. Owing to the low learning outcome, one could fear that increased certainty would lead to an increased risk of error. Therefore, it was satisfying that the overall risk of error declined after the course with both methods. Although a proportion of 22% with an increased risk of error after taking the course seemed alarming, it was within the limit of what could occur by chance, due to the small learning outcome. However, one may speculate that taking courses may increase the risk of error, if the feeling of being secure is increased without a corresponding improvement of knowledge. This might have implications for the need of follow-up after courses.
The factors that were associated with a reduced risk of error after the calculation course could indicate who might benefit from training like this: being a man; working in hospital; low pretest score and low pretest certainty score. This supports the finding in the auxiliary analysis that nurses with weak drug dose calculation skills benefit the most from taking courses. Nevertheless, the risk of error demonstrated in the study did not necessarily reflect the real risk of adverse events affecting patients, as the test situation cannot measure how often miscalculations were performed or how serious the clinical implications might be for any patient. Such studies still need to be done.
Importance for practice
The fact that 48% of the participants in the study performed drug dose calculations at least weekly was more than anticipated. It has been a common perception that the need for most nurses to calculate drug doses is small in today's clinical practice. The reported extent of calculations underscores the importance of good skills in this field.
When the need for continuous improvement and maintenance of skills is identified, the time and resources available will be decisive for the possibility to implement further training activities. E-learning is more often a preferred choice in health services institutions, as it is both flexible and cost-effective. In our study, the e-learning group stated a higher specific value of the course for working situations, although the course content was similar in both methods. However, this method also had more dropouts and a lesser learning outcome for those with low skills. In a review article commenting on the results of a meta-analysis of e-learning and conventional instruction methods, Cook claims that rather than more comparative studies, further research should focus on conditions (how and when) under which e-learning is a preferable method.12
An implication of the findings can be to let nurses regularly attend an e-learning course followed by a screening test to uncover the weak calculation topics. Those who need further training should be offered a more tailored follow-up. Others have also documented that a combination of different learning and teaching strategies do result in better retention of drug calculation skills compared with lectures alone.23 Further studies of the effect of the introduction of drug dose calculation apps would also be of interest, as well as more authentic observation studies in a high fidelity simulation environment, as reported from a Scottish HHS study.26
Interestingly, the e-learning group stated a higher specific value of the course for working situations, although the course content was similar in both methods. This may be explained by the flexibility of the e-learning course, which allowed the participants to concentrate on the items that were considered difficult and relevant for their work, while the classroom group had to follow through the whole programme. Nearly all the nurses themselves realised that they needed more training in drug dose calculations, and an important factor was that motivation for the course was associated with a good learning outcome in the study. This indicates that the professional leadership in health institutions should facilitate and encourage the nurses to improve their skills further in drug dose calculations.
In addition to regularly training in calculations, written procedures for specific dilutions and infusions used in the wards would be of importance as a quality insurance for improved patient safety. This must be a part of the management responsibility.
Study limitations
The participants in this study were recruited through the management line, and the study population represents a limited part of the total nurse population. We assume that nurses with low calculation skills would, to a lesser degree, volunteer for such a study, and hence presume that the calculation skills in clinical practice would be lower than shown in this study. External validity might be an issue in studies with voluntary participation, and extrapolation of the findings of the study to all registered nurses should be performed with caution.
Some may question the quality of the course content and duration or teaching conditions of the courses, especially since the learning outcome of the courses were not convincing. However, the main aim for the study was to compare the two didactic methods. Also, to ensure a fair comparison and similar content of the courses, the subject teacher, who was a part of the group that developed the e-learning course, was also responsible for the classroom lectures. Since the teacher had an interest in both didactic methods, the probability for her to affect the course arrangements in favour of one of them was regarded as small. The questionnaire used was the same as that used to test the nursing students, and the calculation tasks were considered to be in accordance with the tasks that were performed in the nursing practice.
Another limitation could be the controlled test conditions, without time pressure and interruptions that are often the case in a stressful work situation, which tend towards better results than in reality. On the other hand, the calculation test situation itself may be stressful for the nurses, since many have struggled to pass a similar test during their studies.
Selecting two dimensions from the GHQ 30 questionnaire may also be a methodological limitation. Although no correlation between the outcomes and coping or well-being/self-esteem was detected, the usage of only parts of the tool excluded the possibility of detecting an association between physiological well-being in general and drug dose calculation skills.
Conclusion
The study was not able to demonstrate any differences between e-learning and classroom teaching in drug dose calculations, with respect to learning outcome, certainty or risk of error. The overall learning outcome was without practical significance, and conversion of units was the only topic that was significantly improved after the course. An independent factor in favour of classroom teaching was weak pretest knowledge, while factors suggesting use of e-learning could be the need for training in relevant work specific tasks and time effective repetition.
Acknowledgments
The authors wish to thank Innlandet and Oestfold Hospital Trusts and the healthcare administrations of Gjoevik, Hamar and Lillehammer municipalities for letting the nurses participate in the trial during work hours, and the Faculty of Medicine, Norwegian University of Science and Technology for preparing the CRFs for optical reading of the data. The authors thank Stian Lydersen, professor in medical statistics at the Regional Centre for Child and Adolescent Mental Health, Faculty of Medicine, Norwegian University of Science and Technology, Trondheim for statistical advice.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online supplement
Footnotes
-
Contributors BOS was involved in the design making, and was responsible for drafting the study protocol and tests, performing the data collection, and also drafting the analyses and manuscript. IJ supervised the study, and contributed with substantial and useful comments and input during all phases. GKD contributed to the planning of the study tests and data collection, gave valuable input to the interpretation of the results, and participated in the drafting and revisions of the manuscript. PGF was the project leader and supervised the study, and also made incalculable contributions during all phases. All authors approved the published version.
-
Funding The study was funded by research grants from the South-East Norway Health Authorities and Innlandet Hospital Trust.
-
Competing interests GKD was part of the group which developed the e-learning programme used in the study, and the course was made commercially available from autumn 2009.
-
Ethics approval The Norwegian Data Inspectorate, represented by the Privacy Ombudsman for Research.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement The full protocol, the questionnaires and extra data are available by e-mailing the corresponding author: bjorg.simonsen@sykehuset-innlandet.no. The questionnaire is also available in English translation as a supplementary file.