PROGRAM INTEGRITY IN PRIMARY AND EARLY SECONDARY PREVENTION: ARE IMPLEMENTATION EFFECTS OUT OF CONTROL?

doi:10.1016/S0272-7358(97)00043-3

Clinical Psychology Review

Volume 18, Issue 1, January 1998, Pages 23-45

https://doi.org/10.1016/S0272-7358(97)00043-3 Get rights and content

Abstract

We examined the extent to which program integrity (i.e., the degree to which programs were implemented as planned) was verified and promoted in evaluations of primary and early secondary prevention programs published between 1980 and 1994. Only 39 of 162 outcome studies featured specified procedures for the documentation of fidelity. Of these, only 13 considered variations in integrity in analyzing the effects of the program. Lowered adherence to protocol was often associated with poorer outcome. There was mixed evidence of dosage effects. The omission of integrity data, particularly measures of adherence, may compromise the internal validity of outcome studies in the prevention literature. We do not view procedures for integrity verification as inconsistent with the adaptation of interventions to the needs of receiving communities.

Section snippets

THE IMPORTANCE OF INTEGRITY VERIFICATION IN PREVENTION RESEARCH

The verification of program integrity is of particular importance in research into preventive interventions, which are often implemented in conditions that present numerous obstacles to fidelity in program delivery. Prevention researchers, who must often contend with limitations in resources, frequently work with numerous paraprofessionals or lay-professional volunteers as service providers in order to minimize cost and achieve a broad dispersion of preventive services (Institute of Medicine,

THE PROMOTION OF PROGRAM INTEGRITY

Program fidelity has been enhanced through the provision of training manuals (Schinke et al., 1985), the training of facilitators (Connel et al., 1985; Perry et al., 1990; Ross et al., 1991) and the supervision of implementors (Harchik et al., 1992; Peterson et al., 1988). Training manuals with clear descriptions of the activities to be implemented are a key to the promotion of program integrity (Blakely et al., 1987; Dobson and Shaw, 1984; Kazdin, 1986), as well as to its verification and the

THE FIDELITY/ADAPTATION DEBATE

Opinions differ as to the feasibility and desirability of promoting and verifying program integrity in prevention research. Some authors oppose the adaptation of specified program activities by personnel at adopting sites because they believe that program efficacy would be compromised by such changes (Boruch and Gomez, 1977; Calsyn et al., 1977). However, this position may be untenable. The tight prescriptions of program models, often developed without full knowledge of the needs and conditions

THE PRESENT STUDY

According to many sources, the promotion and verification of program integrity should be an important facet of prevention research. However, the prevalence of these features in prevention research has not been examined. Previous reviews of integrity verification practices have focused on school-based behavioral interventions (Gresham et al., 1993), parent-training programs (Rogers-Weise, 1992), outcome studies in clinical psychology, behavior therapy, psychiatry, marital and family therapy (

Verification and Promotion of Program Integrity in the Prevention Literature

As Table 1 illustrates, verification of program integrity appeared in a minority of studies evaluating the outcome of preventive interventions. Promotion of program integrity, however, was somewhat more common. Manuals and training were provided to implementors in slightly more than half of the studies considered; supervision of service providers was indicated in somewhat less than half. However, manuals, training and supervision were provided together in only one fifth of the outcome

DISCUSSION

As described above, the majority of studies in the prevention literature contain no indications of the adequacy of program delivery, or of the number of subjects that participated in a satisfactory proportion of planned sessions. The internal validity of much of the outcome research in this literature may be compromised by these omissions.

Acknowledgements

This research was supported by a research grant awarded to the second author by the Health Canada Strategic Fund for Children's Mental Health.

References (50)

G.L Botvin et al.
A cognitive-behavioral approach to substance abuse prevention: One year follow-up
Addictive Behaviors
(1990)
B.R Flay
Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs
Preventive Medicine
(1986)
F.J Moncher et al.
Treatment fidelity in outcome studies
Clinical Psychology Review
(1991)
J.M Moskowitz et al.
Evaluation of a junior high-school primary prevention program
Addictive Behaviors
(1983)
L.A Rohrbach et al.
Diffusion of a school-based substance abuse prevention program: Predictors of program implementation
Preventive Medicine
(1993)
S Tortu et al.
School-based smoking prevention: The teacher training process
Preventive Medicine
(1989)
J.P Allen et al.
School-based prevention of teenage pregnancy and school drop-out: Process evaluation of the national replication of the teenage outreach program
American Journal of Community Psychology
(1990)
L.J Bauman et al.
Reinventing fidelity: The transfer of social technology among settings
American Journal of Community Psychology
(1991)
P Berman et al.
Implementation of educational innovation
Educational Forum
(1976)
C.H Blakely et al.
The fidelity-adaptation debate: Implications for the implementation of public sector social programs
American Journal of Community Psychology
(1987)

R.R Boruch et al.

Sensitivity, bias, and theory in impact evaluation

Professional Psychology

(1977)

G.J Botvin et al.

Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3 year study

Journal of Consulting and Clinical Psychology

(1990)

G.J Botvin et al.

A skills training approach to smoking prevention among Hispanic youth

Journal of Behavioral Medicine

(1989)

G.J Botvin et al.

Smoking prevention among urban minority youth: Assessing effects on outcome and mediating variables

Health Psychology

(1992)

R Calsyn et al.

Incomplete adoption of an innovation: The case of goal attainment scaling

Evaluation

(1977)

D.B Connel et al.

Summary of findings of the school health education evaluation: Health promotion, effectiveness, implementation and costs

Journal of School Health

(1985)

L.E Datta

Damn the experts and full speed ahead: An examination of the study of federal programs supporting educational change, as evidence against directed development for local problem solving

Evaluation Review

(1981)

K.S Dobson et al.

The use of treatment manuals in cognitive therapy: Experience and issues

Journal of Consulting and Clinical Psychology

(1984)

R.E Drake et al.

Process evaluation in the Washington, DC, Dual Diagnosis Project

Alcohol Treatment Quarterly

(1993)

Elias, M.J., & Clabby, J.F. (1992). Building social problem solving skills: Guidelines from a school-based program. San...

R.D Felner et al.

The parenting partnership: The evaluation of a human service/corporate workplace collaboration for the prevention of substance abuse and mental health problems and the promotion of family and work adjustment

Journal of Primary Prevention

(1994)

R.D Felner et al.

Ecological interventions and the process of change for prevention: Wedding theory and research to implementation in real world settings

American Journal of Community Psychology

(1991)

D.C Gottfredson et al.

Managing adolescent behavior in a multi-year, multi-school study

American Educational Research Journal

(1993)

F.M Gresham

Assessment of treatment integrity in school consultation and prereferral intervention

School Psychology Review

(1989)

F.M Gresham et al.

Treatment integrity of school-based behavioral intervention studies: 1980–1990

School Psychology Review

(1993)

Cited by (1042)

Champion and audit and feedback strategy fidelity and their relationship to depression intervention fidelity: A mixed method study
2023, SSM - Mental Health
Globally, mental health disorders rank as the greatest cause of disability. Low and middle-income countries (LMICs) hold a disproportionate share of the mental health burden, especially as it pertains to depression. Depression is highly prevalent among those with non-communicable diseases (NCDs), creating a barrier to successful treatment. While some treatments have proven efficacy in LMIC settings, wide dissemination is challenged by multiple factors, leading researchers to call for implementation strategies to overcome barriers to care provision. However, implementation strategies are often not well defined or documented, challenging the interpretation of study results and the uptake and replication of strategies in practice settings. Assessing implementation strategy fidelity (ISF), or the extent to which a strategy was implemented as designed, overcomes these challenges. This study assessed fidelity of two implementation strategies (a ‘basic’ champion strategy and an ‘enhanced’ champion + audit and feedback strategy) to improve the integration of a depression intervention, measurement based care (MBC), at 10 NCD clinics in Malawi. The primary goal of this study was to assess the relationship between the implementation strategies and MBC fidelity using a mixed methods approach.
We developed a theory-informed mixed methods fidelity assessment that first combined an implementation strategy specification technique with a fidelity framework. We then created corresponding fidelity indicators to strategy components. Clinical process data and one-on-one in-depth interviews with 45 staff members at 6 clinics were utilized as data sources. Our final analysis used descriptive statistics, reflexive-thematic analysis (RTA), data merging, and triangulation to examine the relationship between ISF and MBC intervention fidelity.
Our mixed methods analysis revealed how ISF may moderate the relationship between the strategies and MBC fidelity. Leadership engagement and implementation climate were critical for clinics to overcome implementation barriers and preserve implementation strategy and MBC fidelity. Descriptive statistics determined champion strategy fidelity to range from 61 to 93% across the 10 clinics. Fidelity to the audit and feedback strategy ranged from 82 to 91% across the 5 clinics assigned to that condition. MBC fidelity ranged from 54 to 95% across all clinics. Although correlations between ISF and MBC fidelity were not statistically significant due to the sample of 10 clinics, associations were in the expected direction and of moderate effect size. A coefficient for shared depression screening among clinicians had greater face validity compared to depression screening coverage and functioned as a proximal indicator of implementation strategy success.
Fidelity to the basic and enhanced strategies varied by site and were influenced by leadership engagement and implementation climate. Champion strategies may benefit from the addition of leadership strategies to help address implementation barriers outside the purview of champions. ISF may moderate the relationship between strategies and implementation outcomes.
Mindfulness-based interventions for stress and burnout in teachers: A systematic review
2023, Teaching and Teacher Education
This systematic review provides an updated synthesis of mindfulness-based interventions (MBIs) for stress and burnout in teachers (K-12) and assessed implementation fidelity. We followed PRISMA guidelines, conducted electronic searches in five databases, and included studies through to February 2022. We included assessments of fidelity (using Treatment Fidelity Tool for MBIs) and methodological quality (using Down and Black checklist). Thirty-nine studies met our inclusion criteria. Evidence of MBIs for reducing stress and burnout, and improving other psychological outcomes showed great promise. However, a lack of standardisation in intervention components, facilitators, duration, and outcome measures was observed. Recommendations for future interventions and research are provided.
Stability and change of secondary school students' motivation profiles in mathematics: Effects of a student intervention
2023, Journal of School Psychology
There is high agreement that motivation is an important factor for successful learning processes and outcomes. But how do students differ in terms of motivation and how do these differences affect the effectiveness of a motivation intervention? As an intervention interacts with students' characteristics, students' heterogeneity must be considered and homogeneous intervention effects must be critically examined. This study aimed to identify motivation profiles of a specifically vulnerable student group, namely students in the lowest ability tier in the learning of mathematics. Within the framework of self-determination theory, we investigated how these profiles changed during Grade 7 and Grade 8. Furthermore, the study examined whether a particular intervention setting aimed at promoting positive emotions and motivation in learning had an impact on the patterns of change in the specific motivation profiles compared to students in the control condition. A latent profile analysis based on self-reported intrinsic, identified, introjected, and external regulation of 348 students revealed three motivation profiles, consisting of (a) low-mixed, (b) high-mixed, and (c) self-determined. Results of the latent transition analysis indicated that the majority of students tended to remain in the same profile and also revealed different effects of the intervention on different motivation profiles. The intervention seemed to be better tailored to students in the low-mixed motivation profile than to students in other profiles. This result highlights the nature of differential effects between students.
Does Binge-Watching eHealth Intervention Content Impact Outcomes?
2023, Journal of Adolescent Health
Advancements in technology have made it possible to deliver parenting interventions online, known as eHealth interventions. Little is known about the rate at which parents participate in eHealth interventions, characteristics of parents who watch eHealth interventions at an accelerated pace (i.e., binge-watching), and if binge-watching impacts intervention outcomes.
The sample included 142 Hispanic parents who were randomly assigned to an eHealth family-based intervention and completed 100% of eight online, prerecorded and self-paced video group sessions delivered across 12 weeks. We examined baseline predictors (parent sociodemographic characteristics, report of child's externalizing behaviors, and family functioning) of watching group sessions in two weeks or less (n = 23, 16.2%). Using latent growth curve modeling, we tested the impact of binge-watching on the trajectory of adolescent drug use, condomless sex, and depressive symptoms across 36 months. We also examined the impact of binge-watching on changes in family functioning from baseline to 6 months postbaseline.
Parents with high levels of education and of children with attention problems were more likely to binge-watch. Conversely, parents of children with conduct disorder symptoms were less likely to binge-watch. The trajectory of depressive symptoms increased for adolescents with parents who binge-watched the intervention, but the trajectory of condomless sex decreased. There was no impact on drug use. Binge-watching was also associated with decreases in parental monitoring.
The findings of this study have implications for eHealth interventions; the pace that parents watch eHealth interventions may subsequently impact adolescent outcomes, such as condomless sex and depressive symptoms.
Implementation and effectiveness of a school-based intervention to increase adherence to national school meal guidelines: A non-randomised controlled trial
2024, Public Health Nutrition
Assessing the implementation fidelity, feasibility, and sustainability of community-based house improvement for malaria control in southern Malawi: a mixed-methods study
2024, BMC Public Health

View all citing articles on Scopus

View full text

PROGRAM INTEGRITY IN PRIMARY AND EARLY SECONDARY PREVENTION: ARE IMPLEMENTATION EFFECTS OUT OF CONTROL?

Abstract

Section snippets

THE IMPORTANCE OF INTEGRITY VERIFICATION IN PREVENTION RESEARCH

THE PROMOTION OF PROGRAM INTEGRITY

THE FIDELITY/ADAPTATION DEBATE

THE PRESENT STUDY

Verification and Promotion of Program Integrity in the Prevention Literature

DISCUSSION

Acknowledgements

Addictive Behaviors

Preventive Medicine

Clinical Psychology Review

Addictive Behaviors

Preventive Medicine

Preventive Medicine

School-based prevention of teenage pregnancy and school drop-out: Process evaluation of the national replication of the teenage outreach program

American Journal of Community Psychology

Reinventing fidelity: The transfer of social technology among settings

American Journal of Community Psychology

Implementation of educational innovation

Educational Forum

The fidelity-adaptation debate: Implications for the implementation of public sector social programs

American Journal of Community Psychology

Sensitivity, bias, and theory in impact evaluation

Professional Psychology

Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3 year study

Journal of Consulting and Clinical Psychology

A skills training approach to smoking prevention among Hispanic youth

Journal of Behavioral Medicine

Smoking prevention among urban minority youth: Assessing effects on outcome and mediating variables

Health Psychology

Incomplete adoption of an innovation: The case of goal attainment scaling

Evaluation

Summary of findings of the school health education evaluation: Health promotion, effectiveness, implementation and costs

Journal of School Health

Damn the experts and full speed ahead: An examination of the study of federal programs supporting educational change, as evidence against directed development for local problem solving

Evaluation Review

The use of treatment manuals in cognitive therapy: Experience and issues

Journal of Consulting and Clinical Psychology

Process evaluation in the Washington, DC, Dual Diagnosis Project

Alcohol Treatment Quarterly

The parenting partnership: The evaluation of a human service/corporate workplace collaboration for the prevention of substance abuse and mental health problems and the promotion of family and work adjustment

Journal of Primary Prevention

Ecological interventions and the process of change for prevention: Wedding theory and research to implementation in real world settings

American Journal of Community Psychology

Managing adolescent behavior in a multi-year, multi-school study

American Educational Research Journal

Assessment of treatment integrity in school consultation and prereferral intervention

School Psychology Review

Treatment integrity of school-based behavioral intervention studies: 1980–1990

School Psychology Review