Article Text
PDF
Abstract
Objectives To determine whether repeated mystery shopping visits with feedback improve pharmacy performance over nine visits and to determine what factors predict an appropriate outcome.
Design Prospective, parallel, repeated intervention, repeated measures mystery shopping (pseudopatient) design.
Setting Thirty-six community pharmacies in metropolitan Sydney, Australia in March–October 2015.
Participants Sixty-one University of Sydney pharmacy undergraduates acted as mystery shoppers. Students enrolled in their third year of Bachelor of Pharmacy in 2015 were eligible to participate. Any community pharmacy in the Sydney metropolitan region was eligible to take part and was selected through convenience sampling.
Intervention Repeated mystery shopping with immediate feedback and coaching.
Outcome measures Outcome for each given scenario (appropriate or not) and questioning scores for each interaction.
Results Five hundred and twenty-one visits were analysed, of which 54% resulted in an appropriate outcome. Questioning scores and the proportion of interactions resulting in an appropriate outcome significantly improved over time (P<0.001). Involvement of pharmacists, visit number, increased questioning score and the prescribed scenario were predictors of an appropriate outcome (P=0.008, P=0.022, P<0.001 and P<0.001, respectively). Interactions involving a pharmacist had greater scores than those without (P<0.001).
Conclusions Repeated mystery shopping visits with feedback were associated with improved pharmacy performance over time. Future work should focus on the role of non-pharmacist staff and design interventions accordingly.
- pharmacy
- simulated patient
- community pharmacy
- minor ailment
- nonprescription medicine
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Large number of repeated mystery shopping visits (n=521).
Examined 10 different minor ailment scenarios.
Thirty-six community pharmacies were mystery shopped.
Pharmacies were restricted to metropolitan Sydney, Australia.
Staff provided consent before mystery shopping visits occurred which may have influenced their handling of requests.
Introduction
It has been suggested that medicines are the most common form of medical intervention in the developed world.1 In recent times, access to increasing numbers of medicines without a prescription is occurring due to the down scheduling of a number of medicines from prescription-only to non-prescription (over-the-counter) status.2 3 These regulatory changes can provide cost savings to insurers and governments,4 5 facilitate patient self-care and self-medication6 and open up greater opportunities for treatment; however, down scheduling can also create potential for medication misadventure.7 8
Pharmacies are important locations for those seeking non-prescription medicines and are equipped to manage a wide variety of ailments.9 10 As pharmacies can play a major role through the provision of medicines and advice giving, it is important that pharmacy staff are adherent to guidelines and provide their patients with evidence-based treatment and advice to ensure optimal health outcomes.
In Australia, the pharmacy workforce consists of pharmacists (either registered or graduate ‘intern’ pharmacists) who have completed a bachelor’s or master’s qualification at university and non-pharmacists (assistants or technicians) whose level of training is not regulated and may vary. As both pharmacists and non-pharmacists are permitted to engage in the sale of medicines and counselling patients on their use, it is important that both parties are appropriately trained to carry out their duties with respect to requests for non-prescription medicines.
Several intervention methods have been employed in the healthcare sector to improve practitioner performance and adherence to guidelines.11 12 Audit and feedback has been identified as a commonly used and effective intervention method to change practitioner behaviour and improve the quality of care provided.13–15 In the context of pharmacy practice, a form of this method of intervention that has been employed is that of mystery shopping with feedback.16 17 This method of intervention has been used in many minor ailment scenarios (vaginal thrush,18 dyspepsia,19 20 asthma21 and headache22) and has been shown to be a feasible and acceptable method of intervention for staff involved.19 23 Further to improving practice, this methodology has also been employed to monitor and audit practice in the pharmacy setting.24 25
This method of intervention involves a mystery shopper (also known as a simulated patient, pseudopatient or secret shopper) entering a pharmacy and undertaking an interaction with a staff member. The mystery shopper should be indistinguishable from a regular patient, generally has a set scenario to follow and may, but not always, be a professional actor.16
In this study, repeated mystery shopping visits with feedback were employed as a coaching technique to improve supply of non-prescription medicines for minor ailments in the community pharmacy setting. Therefore, the aim of this study was to use the method of mystery shopping with feedback to determine whether scores and proportions of individuals achieving an appropriate outcome for each scenario changed over time and to determine what factors predict an appropriate outcome being achieved.
Methods
A prospective parallel, repeated intervention, repeated measures study design was employed to assess the impact of mystery shopping with feedback on community pharmacy practice in Sydney, Australia between March and October 2015.
Ethics
Ethics for this study was submitted to and approved by the Human Research Ethics Committee at the University of Sydney (reference no 2014/186).
Participants and setting
Bachelor of Pharmacy students entering their third year of the degree programme at the University of Sydney in 2015 were invited to take part in the study to be mystery shoppers in lieu of a portion of their regular clinical placements programme. Fifty-nine third-year students consented to take part in the study, with 30 taking part in semester one of the academic year (March–June) and the remaining 29 taking part in semester two (July–November). Two fourth-year honour students shared the role of the remaining place in the second semester to give a total of 61 students participating as mystery shoppers. Thirty-six of 59 third-year students successfully recruited one community pharmacy (n=36) in the Sydney metropolitan region (no other criteria for participating pharmacies were applied other than geographical location). Thirty of these pharmacies were selected to take part in semester one, with the remaining six reserved for semester two. The first 24 pharmacies from semester one to reply to an invitation to take part again in semester two were re-recruited, giving a total of 36 pharmacies taking part across both semesters.
Consent process and training
Authorised representatives from each pharmacy, such as the pharmacy owner or manager, were asked to provide consent for the pharmacy to be used as a location for the study. Each individual staff member at the pharmacy was also invited to provide informed consent, including consent to audio recording, prior to the study commencing. Thus, the study was not truly covert. Pharmacy staff were informed of the timeline of the study and that they would receive one visit each week from a student mystery shopper, but were not informed of the identity of the shopper, which scenario they were allocated or exactly when the visits would occur.
Recruited students provided informed consent to act as mystery shoppers and for audio recording of each visit and feedback session to occur. Students were trained across a 2-day programme where they learnt how to enact all 10 scenarios (of which they went on to shop nine) through role-play with the research team, familiarised themselves with data collection sheets, were taught about theory behind the intervention and trained in providing feedback to pharmacy staff.
Mystery shopping visits and data collection
After completing training, students presented once a week to a preallocated pharmacy for 9 weeks with a scripted direct-product request for a non-prescription medicine, for example, ‘Can I get some Zantac [ranitidine], please?’ Each week individual students visited a different pharmacy in order to minimise the risk of detection. Each pharmacy was allocated a different scenario (1 of 10) relating to a minor ailment. In each semester, three pharmacies were allocated each scenario (10 scenarios×6 pharmacies×9 visits=540 visits in total). Pharmacies that participated in both semesters were allocated a different scenario in the second semester to the one that was allocated in the first, meaning each pharmacy was allocated a maximum of 2 of 10 scenarios across the course of the study. Scenarios included adult cough/cold, adult pain, allergic rhinitis, asthma, diarrhoea, dyspepsia, insomnia, paediatric cough/cold, paediatric fever and smoking cessation. Scenario variables were altered each visit by the researchers in the script given to students, including who the medicine was for, what symptoms they were experiencing, the product requested, if the person was on any other therapy and the legislative status of the product requested. In Australia, medicines available without a prescription are classified into three ‘schedules’. These are Pharmacist Only (a registered pharmacist (or graduate pharmacist) must personally hand the product to the individual requesting it), Pharmacy Medicine (can only be sold under the supervision of a registered pharmacist in a licensed premises) or unscheduled (general sale permitted in outlets such as supermarkets).26 Exemplar particulars for each scenario have been previously published.27
Mystery shoppers requested the product from the first staff member they encountered and followed questioning, counselling and purchasing as directed by the staff member. After purchasing any products (where a product was sold), students exited the pharmacy and completed a scoresheet based on the audio recording of the interaction. Within 5 min, students returned to the pharmacy, provided the staff member with a blank copy of the scoresheet for them to complete as a form of self-evaluation and then provided verbal feedback to the staff member based on their performance. Any products purchased were returned to the pharmacy without refund, and self-evaluation forms were collected from the staff member.
Data collection sheets used in this study (online supplementary file 1) were based on the WHAT-STOP-GO protocol28 developed by the Pharmaceutical Society of Australia to aid pharmacy staff (particularly non-pharmacist staff) in questioning individuals presenting with a request for non-prescription medicines. This aide-mémoire includes the questions Who the medicine is for, How long they have had the symptoms, what the Actual symptoms are and if they are on any other Treatments for this presentation or another condition. The scores for each of these key questions were aggregated and classified as the ‘questioning’ score. For each criterion, a score of yes (2 points), no (0 points), partial (1 point) or not applicable was applied. Scores were then totalled and converted to a percentage to account for any ‘not applicable’ criteria. A maximum questioning score of 18 points was possible depending on the scenario specifics. For example, in the version of the asthma scenario included in the online supplementary file 1, if a staff member asked all seven questions correctly (‘yes’), they would be allocated a questioning score of 14 points, the maximum possible for that scenario. Other data were also recorded on the collection sheets such as the provision of a product and any counselling on its use, referral to a medical practitioner, perceived rapport and the provision of written or verbal information the time and date of the visit, who served the shopper and space for any comments about the interaction. Scores were not calculated for non-questioning aspects of interactions due to the range of possible responses based on the scenario specifics and previous research demonstrating a relationship between questioning and appropriate outcome.18 21
Supplementary file 1
Each scenario version had a given ‘appropriate outcome’ or scenario angle that the pharmacy staff member should have achieved based on the information presented in the scenario (table 1). The research team decided on this outcome, in concordance with current best practice guidelines. For example, in the asthma scenario (online supplementary file 1), the staff member should have identified the patient’s frequent use of salbutamol indicating poorly controlled asthma and appropriately referred the patient to a medical practitioner.29
Example of appropriate outcome for each scenario
Data collation and analysis
Data collection sheets were checked for consistency and completeness and entered into Microsoft Excel 2016 for Windows (Microsoft, Redmond, Washington, USA). A random sample of 10% of recordings were audited by the first author (JCC) to determine the level of discrepancy between the scoring and data collection by the mystery shoppers. This proportion of recordings was selected as the value to audit based on existing literature in this field.30
Data were then imported into IBM SPSS Statistics V.24.0 (SPSS) and descriptively analysed, and tests for normality and homogeneity of data were performed. Pearson’s χ2 analyses were performed to determine whether there was a relationship between pharmacist involvement in the interaction (either by directly serving the shopper or being consulted by a non-pharmacist staff member) and achieving the appropriate outcome, and if there was a difference in the provision of written information between scenarios. Changes in median questioning scores over time were analysed using a Spearman’s rank order correlation. Non-parametric independent-samples median tests were used to ascertain if the median questioning scores significantly differed across scenarios and between pharmacists and non-pharmacists. Point–biserial correlations were performed to determine whether the proportion of interactions resulting in an appropriate outcome differed over the course of the nine visits.
A binary logistic regression model was developed to identify variables that were predictors of a scenario resulting in an appropriate outcome. Independent variables included in this model were whether a pharmacist was involved in the interaction, the questioning score, the visit number (1–9), the legislative status of the product requested, if the mystery shopper was identified by the pharmacy staff, the pharmacy as a whole and each individual scenario. Allergic rhinitis was selected as the reference scenario for the purpose of the model as it had the lowest proportion of appropriate outcome and was therefore deemed the poorest performing scenario. All independent variables were tested for collinearity.
Results
Sixty-one undergraduate pharmacy students completed 540 mystery shopping visits at 36 different community pharmacies across the Sydney metropolitan region from March to October in 2015. Of these visits, 521 (96%) were eligible for analysis. Reasons for exclusion of the remaining 19 visits included: the shopper being identified as a mystery shopper resulting in termination of the interaction (n=8), incomplete or missing datasheets (n=6), consent refused by the staff member (n=3), error by the mystery shopper during the interaction (n=1) and no stock of the requested product (n=1). Four of the excluded visits were from the allergic rhinitis scenario; three each from the diarrhoea, paediatric fever and smoking cessation scenarios; two each from the adult cough/cold and adult pain scenarios and one each from the dyspepsia and insomnia scenarios. Students were identified as mystery shoppers in 6% of cases (n=30). Of these 30 cases, in 22 instances, it was not until the completion of the visit during the staff feedback that the staff revealed to the students that they had suspected they were a mystery shopper.
Outcome and questioning scores over time
An appropriate outcome was achieved in 54% (n=283) of analysed visits. The proportion of visits resulting in an appropriate outcome by scenario is reported in table 2. Point–biserial correlations showed that the proportion of visits resulting in the appropriate outcome increased over the course of the nine visits across all scenarios (rpb=0.192; P<0.001). When examining scenarios individually, an improvement over time was seen in the adult cough/cold scenario (rpb=0.281; P=0.044), the allergic rhinitis scenario (rpb=0.334; P=0.018), the paediatric fever scenario (rpb=0.356; P=0.01) and the smoking cessation scenario (rpb=0.390; P=0.005) (table 2). Table 2 also outlines the proportions of visits resulting in the appropriate outcome by visit clusters (1–3, 4–6 and 7–9) and overall.
Results of overall appropriate outcome over time by scenario
Overall median questioning scores were 44% (range=0%–100%, IQR=22%–75%). Non-parametric independent-samples median tests determined that scores differed significantly across scenarios (P<0.001). Using Spearman’s rank order correlations, questioning scores were found to improve over time (rs=0.204; P<0.001). When examining individual scenarios, questioning scores improved for the adult cough/cold scenario (rs=0.404; P=0.003), the adult pain scenario (rs=0.362; P=0.008), the asthma scenario (rs=0.387; P=0.004) and the paediatric fever scenario (rs=0.430; P=0.002). Table 3 outlines median questioning scores over time for each scenario and pooled scenario data.
Median questioning scores and IQR over time by scenario
Participating pharmacies and pharmacy staff
Half of the participating pharmacies in this study belonged to a chain or banner group (n=18). The majority of the pharmacies were located on a shopping strip (72%), and the remainder were in a shopping mall or similar.
The staff member(s) who interacted with the mystery shopper were recorded in all but one of the eligible visits (n=520/521). A pharmacist was involved in 72% (n=376) of the analysed interactions. The remaining 144 interactions were handled in isolation by a non-pharmacist staff member. Interactions without the involvement of a pharmacist resulted in the appropriate outcome being achieved in 33% of cases, whereas interactions with a pharmacist resulted in the appropriate outcome in 62% of cases. Pearson’s χ2 analyses found this to be a significant difference (χ2=35.04; P<0.001).
Median questioning and total scores were also found to be significantly different between interactions with a pharmacist and those without (P<0.001). The median questioning score for interactions with a pharmacist was 50% (range=0%–100%, IQR=31%–75%) vs 25% (range=0%–100%, IQR=11%–54%) for those without.
Regression model
The binary logistic regression model is shown in table 4. The model returned a Nagelkerke R2 value of 0.60. Involvement of the pharmacist in the interaction, the visit number, the questioning score and the scenario type were all found to be significant positive predictors (P=0.008, P=0.022, P<0.001 and P<0.001, respectively) of achievement of the appropriate outcome, that is, providing an ‘appropriate’ outcome for the mystery shopper. The legislative status of the product requested, if the shopper was identified by pharmacy staff and the individual pharmacy were not found to be significant predictors. As the pharmacy was not found to be significant, this category was not broken down further.
Factors predicting appropriate outcome of mystery shopping scenarios—binary logistic regression model output (Nagelkerke R2=0.60)
Discussion
This study is the first to use the mystery shopping with feedback methodology across a large number of minor ailment scenarios with multiple repeated visits. The results from the 521 eligible visits demonstrated that multiple visits with feedback were associated with improvement in both the questioning scores of the pharmacy staff participants over time, as well as the proportion of visits achieving an appropriate outcome.
The apparent success of this intervention may be explained primarily by two factors: (1) the use of feedback as an interventional method and (2) the anticipation of a mystery shopping visit. Existing literature in the area of audit and feedback,13–15 in particular Kluger and DeNisi’s Feedback Intervention Theory (FIT),31 supports this hypothesis. This theory postulates that individuals compare their behaviour to the standard that is expected and when they identify that there is inconsistency between the two, they alter their behaviour in order to achieve this benchmark.14 The purpose of audit and feedback is to provide a means through which individuals can identify areas of their practice that do not meet expected standards, while also providing information on how to alter their focus and improve behaviour.14 Several factors that have been identified through the FIT and meta-analyses of audit and feedback intervention studies14 to be positive predictors of a successful audit and feedback intervention were used in this study. These include information about how to perform the task correctly, written feedback, monitoring change from a previous time period and the provision of individual feedback.14 Future mystery shopping interventions may wish to include other design aspects that have also been positively associated with successful interventions such as individual and group feedback, the provision of graphic materials and goal setting.14 Although FIT may explain the apparent success of the intervention, the dynamic between staff and students must be considered. The quality of the feedback provided by the students and the acceptability of this feedback to staff members have not yet been explored. Future work may wish to examine student feedback quality and acceptability. The provision of feedback to individual staff, as opposed to the entire pharmacy, does mean that the study design relies on staff disseminating the information between themselves or the same staff member being shopped multiple times in order for the intervention to be effective. Despite this, the facilitated self-reflection employed in this study through the use of the scoresheets given to staff after the interaction has been shown to be a powerful tool in a previous mystery shopping intervention study.23
The second factor suspected to contribute to the apparent success of the intervention is the notion that the staff knew the mystery shopping programme was taking place and were anticipating a mystery shopping visit. It is speculated that this anticipation generates a Hawthorne effect whereby the staff enter a state of ‘hypervigilence’ and practice at optimal levels due to the possibility that they could be assessed by a mystery shopper at any time.
It is interesting to note that while there was an overall improvement over time when the data are pooled, it was not consistent between scenarios. Further investigation is warranted in targeting the scenarios that performed less favourably to determine why this may have occurred. It is possible that it may be either a result of the nature of the scenarios or the way they were designed by the research team. The scenarios were all designed as direct-product requests, rather than symptom-based requests. Direct-product requests have previously been shown to be negotiated more poorly by pharmacy staff,22 32 33 which may explain the poor results seen in some scenarios, but does not explain interscenario differences.
Existing literature in this field reports contradictory results in regard to a pharmacist being involved in the interaction and a correlation with a higher questioning score and subsequently an appropriate outcome.11 22 34 Previous mystery shopping intervention studies have also identified a correlation between the number of questions asked and a successful outcome, both in the provision of salbutamol for asthma21 and antifungal medications for vaginal thrush.18 Despite increased questioning correlating to an appropriate outcome, research has suggested that not only is any questioning important, but it is also important to consider the types of questions asked and if these questions lead the staff member to an appropriate outcome.5 35 It has been suggested that protocols, such as WWHAM (Who is the patient, What are the symptoms, How long have they had the symptoms, Medicines tried already for the presenting symptoms or taken regularly) in the United Kingdom,36 may not be sufficient in isolation to take an accurate history, arrive at the correct diagnosis and make an appropriate recommendation, and the use of questions targeted at the specific patient and scenario may be more likely to elicit a desirable outcome.35 36 It is interesting to note that in an Australian context, the WHAT-STOP-GO protocol is primarily targeted at non-pharmacist staff. Considering the lack of clinical background and speciality in medicines, the role of these protocols in training both pharmacists and other staff should be evaluated. A recent study has suggested that encouraging patients to ask the pharmacist more questions about their medicine may result in the provision of more information and a longer consultation.37 The feasibility of facilitating patient engagement with their pharmacist when requesting non-prescription medicines should be explored further.
It is worthy to note the difference in results between each of the scenarios. To assess the efficacy of this intervention, scenarios were designed to be of a range of difficulties. In a separate study, using 3 of 10 scenarios in this dataset, the scenarios were designed explicitly to elicit mystery shopper referral to a medical practitioner (asthma, diarrhoea and dyspepsia) and were not found to improve over time.38 This may be due to these scenarios being more difficult and the role of referral not being engrained in day-to-day practice. The scenarios that performed most poorly in this study, allergic rhinitis and smoking cessation, may also be due to the difficult nature of the scenario design. In the smoking cessation scenario, staff were required to identify an interaction between recent cessation of smoking and the decreased cytochrome P450-mediated metabolism of caffeine,39 whereas in the allergic rhinitis scenario, staff were presented with a request for an antihistamine and expected to ‘step up’ the patient to an intranasal corticosteroid as recommended by current guidelines.40 Despite the initial low scores of these scenarios, improvement was seen over time with repeated mystery shopping visits and feedback. It is important to note that this study did not examine if the staff made inappropriate recommendations, but whether or not the staff achieved the ‘gold standard’ outcome as determined by the pharmacist research team when writing the scenarios. Due to variability in scenario performance, particularly in these more difficult scenarios, future interventions may want to focus on specific topic areas. Further work is also needed to determine why pharmacy staff are performing differently in these situations.
The strengths of this study lie in the large number of visits included in the analysis and the variety of scenarios used. However, this study is limited by several factors. The nature of the mystery shopper methodology means that it is never truly possible to determine what happens with real patients and how interactions with pharmacy staff may shape their health outcomes in the future. Instead, surrogate markers such as an appropriate outcome must be relied on. Despite the large number of visits, these were restricted to the metropolitan region of Sydney and may not be generalisable to the rest of Australia or the world. Although staff identification of the mystery shopper was not frequently reported, it is possible that this was under-reported and this may have impacted how the staff handled the mystery shoppers’ requests. Likewise, the staff were aware that the study was taking place, and despite the staff not knowing the exact timing of the visits, the likelihood of future visits would have become apparent after the initial visit and feedback session. Pharmacies were also able to keep the cost of any sales made to the mystery shoppers which again may have influenced practice. The datasheet used in this study has not been validated for use in this setting, and despite a proportion of recordings being audited against the recorded scores, it cannot be guaranteed that the written recording of data by student mystery shoppers was completely accurate in all cases. The voluntary nature of participation for the pharmacies may also have resulted in the pharmacies that are already more likely to perform better. Limitations in the ability to identify whether a single staff member was involved in a mystery shopping interaction on multiple occasions and exactly who served the mystery shopper at first must also be considered when interpreting the results of this study.
Conclusion
Mystery shopping with feedback across multiple visits were associated with improved pharmacy staff performance over time. Multiple visits, pharmacist involvement, increased questioning and the prescribed scenario were predictors of a visit resulting in an appropriate outcome. Mystery shopping with feedback should be explored as a means to train pharmacy staff in an appropriate provision of non-prescription medicines to ensure optimal patient outcomes. Success in the field of pharmacy may warrant exploring the use of this methodology in other healthcare settings. Future interventions should focus on the contribution of non-pharmacist staff to patient care and design interventions to better target this population.
Acknowledgments
The authors wish to acknowledge all of the Bachelor of Pharmacy students and pharmacy staff who participated in this study.
References
Footnotes
Contributors CRS, ACdAN and RJM conceived the study design. CRS, FW and RJM coordinated the study. JCC, CRS, FW, ACdAN and RJM trained students. FW designed scenarios. JCC and CLN completed data collection and data entry. JCC, CRS and RJM conducted data analysis and interpretation of results. JCC drafted the manuscript. All authors contributed to the critical revision of the manuscript and approved the final manuscript for submission.
Funding This work was supported by the Australian Government Office for Learning and Teaching (grant number SD14-4207); the Pharmacy Council of New South Wales (grant number G176927) and Alphapharm Pty Ltd (grant number G176862).
Competing interests None declared.
Ethics approval The University of Sydney Human Research Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data available.