Article Text
PDF
Abstract
Background The positive effects of the Dutch Community Occupational Therapy in Dementia programme on patients' daily functioning were not found in a multicentre randomised controlled trial (RCT) in Germany.
Objectives To evaluate possible effect modification on the primary outcome within the German RCT with regard to (1) participant characteristics, (2) treatment performance and (3) healthcare service utilisation; and (4) to compare the design and primary outcome between the German and the original Dutch study.
Methods (1) The impact of participant baseline data on the primary outcome was analysed in exploratory ANCOVA and regression analyses. (2) Therapists completed questionnaires on context and performance problems. The main problems were identified by a qualitative content analysis and focus-group discussion. Associations of the primary outcome with scores of participant adherence and treatment performance were evaluated by regression analysis. (3) Utilisation rates of healthcare services were controlled for significant group differences. (4) Differences in the Dutch and German study design were identified, and the primary outcome was contrasted at the item level.
Results (1) Participant characteristics could not explain more than 5% of outcome variance. (2) The treatment performance of some active intervention components was poor but not significantly associated with the primary outcome. (3) There were no significant group differences in the utilisation of healthcare resources. (4) In contrast to the Dutch waiting-control group, the active intervention in the German control group may have reduced group differences in the current RCT. The German patients demonstrated a higher independence at baseline and less improvement in instrumental activities of daily living.
Conclusion The differences in outcome may be explained by a more active control treatment, partially poor experimental treatment and less room for improvement in the German sample. Future cross-national transfers should be prepared by pilot studies assessing the applicability of the intervention and patient needs specific to the target country.
Trial registration International Clinical Trials Registry Platform, DRKS00000053.
- RCT—Alzheimer's disease—complex intervention—process evaluation
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Statistics from Altmetric.com
Article summary
Article focus
Lessons to be learnt from a process evaluation alongside a pragmatic multicentre randomised controlled trial, specifically on the effectiveness of a Dutch community occupational therapy in Dementia programme within the German healthcare system, and more generally for cross-national transfer studies.
Key messages
Exploratory analyses of process data did not reveal a major bias within the German trial from the specific baseline characteristics of the participants, variances in the treatment performance or the utilisation of other healthcare resources during the treatment period.
Compared with the original Dutch study, the German sample showed more independence at baseline in items of instrumental activities of daily living and less improvement within an identical treatment period. The Dutch waiting-group design versus the German multicentre parallel group design with an active control arm, as well as the extensive training and expertise of the Dutch therapists might also help to explain the differences in the results from the Dutch and the German randomised controlled trial. To address these findings, future cross-national transfers of complex interventions should be prepared by small-scaled pilot studies assessing the applicability of the intervention, the appropriateness of inclusion criteria and the specific patient needs in the target country.
Strengths and limitations of this study
The association of variance in treatment performance and primary outcomes was investigated alongside a German randomised multicentre trial. Data on the primary outcome were available at the item level from both German and Dutch original studies.
The exploratory statistical analysis of change to baseline in daily functioning in the German sample was based on data which showed variance comparable with the Dutch sample but were possibly limited by a restricted range at baseline. The quality of the treatment performance was measured only by the therapists' self-report.
Introduction
New guidance from the British Medical Research Council states that developing and evaluating complex interventions can be a lengthy process. All steps should be sufficiently addressed. These steps include (1) the development of the intervention, (2) a pilot study on feasibility, (3) a randomised controlled trial (RCT) on effectiveness and (4) an evaluation of implementation in healthcare practice.1 Cross-national transfer of complex intervention can speed up the uptake of innovative and effective programmes from one country to another. Time and resources might be saved when an intervention programme that has already been developed, piloted and evaluated on effectiveness in one country can be directly proven regarding effectiveness in the healthcare context of another country. We followed this approach by transferring the Dutch evidence-based Community Occupational Therapy in Dementia Programme (COTiD)2 to the German healthcare system and testing its effectiveness in a seven-centre RCT.3 However, the highly positive effects of the Dutch COTiD on patients' daily functioning could not be found. Process evaluation is recommended as being highly valuable in RCTs to provide an insight into unexpected intervention failure.4 Differences in participants as well as aspects of treatment performance and contextual factors should be assessed with regard to their associations with the primary outcome.5–7 Based on these recommendations, our process evaluation investigated four research questions. We evaluated both possible bias within the German study (question 1–3) and differences between the Dutch and German RCT (question 4).
Did specific patient or carer characteristics influence a patient's outcome after the intervention?
What problems and variations in experimental treatment performance could be identified in the study context, and did they influence the daily functioning of patients?
What differences in the utilisation of further healthcare resources during the treatment period could be identified, and did they influence the daily functioning of patients?
What differences between the Dutch and the German study could be identified in terms of design and primary outcome?
Methods
Specific participant characteristics of the German sample
The outcome of interest was daily functioning, indicated by two measurement instruments, the Interview for Deterioration in Daily Living Activities in Dementia (IDDD) and the Perceive, Recall, Plan and Perform System of Task Analysis (PRPP). The IDDD performance scale records the patients' need for assistance in 11 basic and instrumental activities of daily living.8 In the PRPP, the number of errors occurring during the performance of a self-chosen daily living task is measured.9 An ANCOVA was used to investigate the mean changes from baseline in the IDDD and PRPP between the COTiD and control group controlling for (1) the patient's age, gender, education and financial limitation; and the daily activities, mood and cognition at baseline; and (2) the carer's gender, education, relationship to the patient; and the sense of competence and mood at baseline. Data were collected with standardised measurement instruments as described in the study protocol and were in line with a recent health-technology assessment of risk or protective factors for Alzheimer's disease.10 11 The percentage variance explained in mean changes from baseline in the IDDD and PRPP was assessed using multiple regression.
COTiD performance in the German experimental group
Therapists completed semistructured questionnaires during and after the treatment period. (Questionnaires are available in German from the corresponding author.) During the treatment phase, they reported reasons for a problematic performance in 20 subprocesses for each experimental case. The subprocesses were defined according to the study protocol (table 1 and 4). After the treatment period, therapists described and rated their professional experience in the field and their valuation of introduction, pilot phase and supervision, as well as inhibiting and facilitating processes at the study site. A qualitative content analysis with inductive category development was used to identify the main performance problems from the comments given in the questionnaires.12–16 A focus-group discussion served as a member check, in order to achieve consensus among the therapists about the main performance problems.17–21
Statements by therapists stating main performance problems within the therapeutic subprocesses
Furthermore, the therapists dichotomously scored the 20 treatment subprocesses as performed either with or without problems. These scores were used to operationalise the quality of performance for each case. The best quality was indicated by 100% when all subprocesses were performed without any problems.
The therapists also rated patient adherence regarding the cooperation during the interview, the goal setting and the training; as well as regarding the patient's daily changing mental capacity, their collaboration with the carer and regarding the acceptance of innovations. Additionally, the carer adherence was assessed with regard to their cooperation during the scheduling, the interview, the goal setting and the training of supervision, as well as with regard to their encouragement of the patient, the acceptance of support service and the implementation of innovations. Therapists rated these indicators for adherence on a five-point-Likert scale ranging from ‘very facilitating for the treatment performance’ (=1) to ‘very hindering for the treatment performance’ (=5).
Correlations between the mean changes from baseline in the IDDD and PRPP and scores of the performance quality and the participant adherence were calculated (Pearson coefficient). An exploratory regression analysis was deemed to be appropriate for smaller samples, and this was to evaluate whether such scores could explain variance in the mean changes from baseline in the IDDD and PRPP.22
Utilisation of healthcare resources in the German study
The Resource Utilisation in Dementia23 was applied to collect patient data during the treatment period on (1) the number of consultations with general practitioners, neurologists/psychiatrists and other medical specialists; (2) the time for individual therapy such as physio-, speech- or psychotherapy; (3) the time for group therapy such as cognitive stimulation or exercise groups; (4) times of receiving nursing or domestic home care; (5) the number of technical aids implemented within the patient's home; and (6) increasing, decreasing, constant or no intake of acetyl cholinesterase inhibitors. Furthermore, comorbidity indicating a possible need for further healthcare services was rated using the Cumulative Illness Rating Scale.24 These data were tested for the significance of group differences between the experimental and the control arm (non-parametric two-tailed Mann–Whitney U test owing to the negative skewness of the data distribution).
Comparison between the Dutch and the German study
Regarding the patient characteristics and change from baseline to follow-up on the primary outcome, we compared the single IDDD items of the Dutch and the German sample within the identical measurement period of 5 weeks from baseline to the first follow-up measurement at week 6. This was in order to assess whether both samples had the same room for improvement in items which indicate the need for assistance in daily activities. Furthermore, we compared the expertise of the Dutch and German therapists in terms of pre-experience with the experimental intervention and intensity of treatment delivery (patients per therapist) and the study designs regarding the control-group intervention.2 3
Results
Specific participant characteristics of the German sample
The mean changes to baseline in the IDDD and the PRPP were associated neither with carers' socio-demographic or baseline assessment data nor with patients' socio-demographic data or baseline mini mental state (table 2). We found a minor correlation of mean changes to baseline in the IDDD with patients' mood at baseline (Cornell Scale for Depression in Dementia (CSDD)25; r=0.21; p=0.044). A stepwise regression analysis using patient and carer characteristics as listed in table 2 could not explain more than 5% of variance in change over time in the patient's daily functioning. An adjusted ANCOVA using the patient's baseline values of the CSDD, the PRPP and the IDDD as independent variables did not yield any significant group differences in the dependent variables, which were the mean changes to baseline in the IDDD and the PRPP (results not shown). This indicated that after correction for baseline scores of mood and daily functioning, there were still no significant differences on the primary outcome in the German sample with moderate to good daily functioning at baseline.
Pearson correlation coefficient of specific participant characteristics and mean changes to baseline in the Interview for Deterioration in Daily Living Activities in Dementia and the Perceive, Recall, Plan and Perform System of Task Analysis (German completers of the Community Occupational Therapy in Dementia Programme and control group)
COTiD performance in the German experimental group
Eleven therapists from seven study sites delivered the COTiD to 54 patients. The therapists' characteristics (table 3) varied in previous years in dementia care from 1 year part time to 11 years full time, in perceived facilitators from quite facilitating to slightly hindering and in the quality of treatment performance from 52 to 90% of optimal performance. The data did not provide stable patterns in the sense that many previous years in dementia care and high values of perceived facilitators did lead to a high quality of treatment performance or vice versa.
Characteristics of the 11 therapists who delivered Community Occupational Therapy in Dementia Programme to 54 patients with Alzheimer's (German completers of the experimental intervention)
The quality of the subprocess performance (table 4) did also vary from receiving full medical information in 52 of 54 cases (96%) to successfully adapting physical environment in 24 cases (44%). Subprocesses relating to therapeutic active agents as identified by Graff et al15 were performed with no problems at only a low frequency with 76% for setting therapy goals, 46% for training of patient's skills, 44 and 46% for adapting physical and social environment, and 59 and 54% for training of carer's competence in instruction and problem-solving. In the questionnaires, the therapists commented on the main performance problems in these subprocesses (table 4).
Quality of subprocesses of Community Occupational Therapy in Dementia Programme performance in 54 Alzheimer's disease patients (German completers of the experimental intervention)
Association between COTiD performance and primary outcome
We found no significant associations between the scores of COTiD performance and changes to baseline in the IDDD and in the PRPP (detailed data not shown). Since there was a poor performance in those subprocesses which were related to active therapeutic agents (nos 14 to 20; table 4), we further analysed the association between the performance score of these subprocesses and the changes to baseline in the IDDD and in the PRPP. A minimal correlation was found, r=0.268 (p=0.05) only with the PRPP. No association was found between carer adherence and the changes to baseline in the IDDD and in the PRPP. The score of patient adherence and the change to baseline in the PRPP demonstrated a moderate correlation of r=−0.317 (p=0.02). The subsequent regression analysis revealed that patient adherence could explain 10% of the variance (p=0.02) in the PRPP change to baseline. The IDDD change to baseline could not be explained by patient or carer adherence, or by the quality of treatment performance.
Utilisation of healthcare resources in the German study
The COTiD group had a somewhat higher comorbidity index, slightly more visits to general practitioners and fewer hours for nursing or domestic home care. Negative skewness in the data distribution indicated that many participants had a low utilisation rate, and only a few participants had a high intensity of resource utilisation (table 5). However, we found no significant differences on group level within the German trial in any resource utilisation or comorbidity. The subgroups of patients with decreasing or increasing acetylcholinesterase-inhibitor medication were too small to detect any significant group differences. However, the daily functioning in the COTiD group was not better than in the control group, although more COTiD patients received acetylcholinesterase inhibitors at a constant level (COTiD, 63% vs control, 52%).
Utilisation of further healthcare resources of patients with Alzheimer's during the intervention period of intense occupational therapy compared with a single session control intervention (German completers of the Community Occupational Therapy in Dementia Programme and control group)
Comparison between the Dutch and the German study
Differences in the room for improvement in the IDDD
Table 6 shows that the COTiD group in the Dutch sample did improve notably in household instrumental activities of daily living (IADL), only marginally in basic activities of daily living and not at all in handling finances. Graff et al2 defined 20% improvement as being clinically relevant, which is indicated by a pre–post-treatment difference of 0.8 on item-level. The household IADL items demonstrated such differences and, therefore, a high responsiveness to the COTiD programme. Thus, the household IADL items can be presumed to be a therapeutic window basically providing room for improvement given a sufficient need for assistance in these items at baseline. Comparing the Dutch and German COTiD groups, the baseline values in these IADL items differed considerably more than in the other IDDD items.
Responsiveness of specific activities of daily living after an identical treatment period of 5 weeks in the Dutch and the German sample
The German patients showed much less need for assistance in this area. The limited room for improvement in the German sample is obvious when regarding the baseline differences between the Dutch and the German sample. Analysis of a German subsample matched to the Dutch sample with a comparable need for assistance in these household IADL items at baseline was not possible owing to the low number of German patients with such baseline values.
Differences in design
The German trial design included a comprehensive consultation as active control intervention which approximately represents the non-pharmacological standard care in Germany. This was in order to evaluate the possible benefits of COTiD in addition to standard care. Compared with the waiting-control-group design of the Dutch original trial, the German active control intervention may have reduced the group differences in daily functioning after the treatment. Compared with the Dutch therapists, the therapists in Germany had less experience with COTiD before their study involvement (NL: 240 h vs GER: 0 h), less seminar and training time in the study preparation phase (80 h vs 40 h) and fewer COTiD patients per therapist during the treatment period (34 vs 5).
Discussion
The process evaluation of our multicentre RCT on community occupational therapy in Alzheimer's disease revealed that the characteristics of the German participants at baseline did not mediate patients' daily functioning after treatment as indicated by the mean change to baseline in the IDDD and PRPP. Some subprocesses, which were deemed to be active components of the applied complex psycho-social experimental intervention, were performed poorly. However, variances in the performance were not associated with patients' mean change to baseline in the IDDD and PRPP. The utilisation of further healthcare resources was equal in the experimental and control groups. Based on exploratory analyses of process data, we can reject the hypothesis that group differences in participant characteristics and variances in the treatment performance or the utilisation of further healthcare resources had a confounding influence on the primary outcome within the German study sample. The analyses were limited by the restriction of range in the IDDD baseline data within the German sample. However, the variance of the IDDD baseline data in the German sample was higher than in the Dutch sample (German sample: experimental group, mean 15.4 (SD 9.9); control group, 14.1 (10.1); Dutch sample: experimental group 23.5 (7.9); control group, 24.5 (8.7)).
Using the same eligibility criteria, the German sample showed much less room for improvement in daily functioning than the Dutch sample. In Germany, patients' daily functioning at baseline was much better. Most German patients still performed better at the end of the study irrespective of group assignment than the successfully treated Dutch group, in which patients had lower baseline scores and improved significantly. This underlines the importance to pay attention to the needs of the patients and care givers specific to the target country.
In the German study, the self-reported performance of active intervention components was not associated with the primary outcome. The small sample size and the method of self-rating are limitations for detecting such associations. Although an exploratory regression analysis is vulnerable to misinterpretation,22 we also performed this type of analysis, in order to detect any signs of an influence of treatment performance on the primary outcome. However, we found only minor rates of correlation and explanation, which makes any meaningful association between variances in the treatment performance and the primary outcome unlikely. Self-rating can be a feasible approach for evaluating adherence in dementia research,28 in order to deal with limited resources.29 However, therapists tend to overestimate their own performance.30 Although the therapists were explicitly asked to be critical when judging their own performance, for further studies it is recommended that there be an additional external monitoring of treatment performance. This may reduce possible bias introduced by overestimation or overcriticism in self-rating. Furthermore, it might help to find appropriate onsite coaching strategies. These strategies should aim at high-quality treatment performance, even though the complexity of psycho-social interventions induces variance—especially in multicentre RCTs.
Data on the association of treatment performance and primary patient outcomes, although encouraged,5 31 are scarce in RCTs studying complex interventions. Teri et al28 implemented an external video rating of therapists' adherence to protocol but found no associations between this rating and any outcome variable. Similar studies in the field32 33 did not operationalise the impact of treatment performance on the primary outcome as the British Medical Research Council had strongly recommended.1 Within the original Dutch RCT, the large number of patients treated by two highly motivated therapists from the same study site suggests an excellent quality of treatment performance.2 However, its association with the patients' outcome was not quantified. The experiences with the subsequent Dutch implementation of the guidelines revealed that novices had difficulties in adapting to this highly complex intervention in a concrete treatment setting. Prior to cross-national implementation of complex interventions, a successful national transfer from a single-centre setting with highly motivated specialists to a multicentre routine setting with therapists varying in competence and motivation seems to be appropriate.
Conclusion
Our process evaluation revealed that the participants in the study may have had insufficient need for the applied treatment and that active components of the complex psycho-social intervention were poorly performed. Also, an interaction can be considered in the sense that little need for assistance can make a less intensive, one-session treatment appropriate, as applied to the control group. These recent experiences suggest that cross-national transfers are best prepared by a pilot study in the target country exploring specific patient needs, the feasibility of inclusion criteria, the usability of measurement instruments and the applicability of the complex intervention by therapists in routine care settings.
Acknowledgments
We thank all participants and therapists for their contribution. We acknowledge C Mayers, Research Fellow, Faculty of Health & Life Sciences, York St John University, UK, for critical reading and English correction.
References
Footnotes
To cite: Voigt-Radloff S, Graff M, Leonhart R, et al. Why did an effective Dutch complex psycho-social intervention for people with dementia not work in the German healthcare context? Lessons learnt from a process evaluation alongside a multicentre RCT. BMJ Open 2011;1:e000094. doi:10.1136/bmjopen-2011-000094
Funding German Federal Ministry of Health, Reference Number: IIA5-2508FSB111/44-004.
Competing interests MOR: consultancy for Jansen-Cilag and Numico. MH: grants from Wyeth, Pfizer and Medivation; payment for lectures from Wyeth, Pfizer and Merz.
Ethics approval Ethics approval was provided by the Medical Ethics Committee of the University Hospital Freiburg (no 110/08).
Contributors SVR, MG, RL, MH and MVD contributed to the study conception and design. SVR conducted the data and study management, and prepared the statistical analysis. RL performed the statistical analysis. SVR drafted the manuscript. MG, RL, MH, MOR and MVD revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data sets can be provided on request for fellow researchers in the context of collaborative projects and publications.