Evaluation of a complex, multisite, multilevel grants initiative
Introduction
The goal of the Safe Schools/Healthy Students (SS/HS) initiative is to foster integrated systems that create safe and respectful school climates and, consequently, promote the mental health of students and prevent violence and substance abuse. Toward this end, the SS/HS initiative provides funding to local education agencies (LEAs). These LEAs are charged to work in partnership with local law enforcement, juvenile justice agencies, social service and mental health agencies, and other community organizations to develop and implement comprehensive SS/HS plans to achieve the program goal. It is expected that in implementing their comprehensive plans, SS/HS grantees will transform their service delivery systems and create an integrated network of activities, programs, services, and policies that:
- •
Promote the healthy development of social and emotional skills in early childhood.
- •
Enhance prosocial behaviors to prevent violent behavior and drug use.
- •
Increase availability of mental health services.
- •
Reduce incidents of violent behavior and drug use.
- •
Create schools and communities that are safe, disciplined, and drug free.
- •
Engage parents, community organizations, faith-based groups, and other agencies.
- •
Develop an infrastructure that is institutionalized and sustained after the federal funding ends.
Among federal government programs for school-aged youth, the SS/HS initiative is unique in applying a collaborative approach to intervention and service delivery for this population. Like many public sector and nonprofit initiatives, the SS/HS initiative encourages use of best practices and evidence-based programs while promoting changes in the service delivery system. However, the SS/HS grants are also meant to counter the effects of funding and organizational “silos”. By requiring multiple providers at each site to collaborate on improving the systems that serve school-aged youth, Congress and the sponsoring federal agencies (the Departments of Education, Health and Human Services, and Justice) anticipate that SS/HS grantees are more likely to achieve improvement in overall school climate and, consequently, contribute to a variety of individual-level outcomes among school-aged youth.
The SS/HS initiative's focus on creating a safe and respectful school climate as a foundation for promoting the mental health of students and preventing violence and substance abuse is derived from decades of observations of the effect of school climate on student outcomes. Freiberg and Stein (1999) described school climate as “the heart and soul of a school… that quality of a school that helps each individual feel personal worth, dignity, and importance while simultaneously helping create a sense of belonging to something beyond ourselves” (p. 11).
The effects of school climate on youth outcomes have been systematically documented since the 1980s. These observations began with recognition that overall school climate appeared to have a decisive impact on the attainment of academic achievement (Anderson, 1982, Gregory and Smith, 1987, West, 1985); an often-cited U.S. Department of Education study based on data from over 600 secondary schools identified the association of school climate with behavioral problems and dropout rates (Gottfredsen & Gottfredsen, 1989). This association has been confirmed in recent years with increasingly large and diverse data sets, both in the United States and abroad (cf., Khoury-Kassabri et al., 2005, Portes and Hao, 2004). Findings from this research should not lead to the conclusion that low academic achievement, substance abuse, violence, and mental health problems are generated by school climate. Nevertheless, the findings indicate that, in the long term, the efficacy of efforts to address these issues is strongly influenced by school climate.
Fostering effective interventions within the context of improved overall school climate has been recognized as a challenge for two reasons. First, there has been a lack of consensus on the precise definition of “school climate” and the components of measures that assess school climate as a variable (Anderson, 1982). Second, the coordination required to influence overall school climate has been frustrated by the tendency of school-related expertise to be compartmentalized within specific professional fields. For example, innovations in instructional techniques that influence school climate are developed and championed among the teaching community, while innovations in school structure and organization that also influence school climate tend to be developed and championed among experts in educational administration; the specialization and alternative educational patterns of these two groups of professionals tends to create a lack of communication and shared governance (Monk, 2008). The situation is aggravated by the involvement of additional professionals who also influence school climate, such as law enforcement and juvenile justice specialists addressing crime and violence, mental health professionals addressing emotional and mental health issues, and substance abuse specialists addressing alcohol and drug concerns. This complex of youth-related expertise is divided organizationally and fiscally at the local, state, and national levels among agencies with distinct missions and responsibilities, generating separate and parallel planning and funding silos.
Nearly three decades ago, Jessor and Jessor (1977) criticized the “silo” approach to interventions that causes financial and programmatic support to target specific behaviors such as adolescent substance use and abuse, early aggression and violent behavior, and sexual experimentation. Within the context of their Problem Behavior Theory, Jessor and Jessor advocated that more attention and programmatic effort be paid to a wider array of highly interconnected problem behaviors in children and youth. Although researchers and practitioners subsequently developed prevention and intervention models that incorporate this perspective, few have made a determined effort to replace the “silo” model of service planning and delivery with a more collaborative structure. Positive Behavioral Support, for example, an alternative paradigm to addressing multiple problem behaviors, nevertheless emphasizes changes within the school structure and assumes that service delivery among other providers will be integrated for individual high-risk youth (cf., Luiselli et al., 2005, McIntosh et al., 2006).
Whelage and White (1995) challenged approaches that place priority on making services “respond more efficiently or effectively to individual clients or customers”. They claimed such reforms inhibit the formation of “social capital” within a service community. Whelage and White argued that in a true collaboration, the service planning and providing agencies should have a separate form of governance that expands social capital to most effectively use available resources for the benefit of the entire community, rather than specific individuals. According to Whelage and White, “In organizing themselves toward common purposes, people use communications networks and draw upon the trust that has been established within networks. …Trust and reliable communication networks serve as the glue that holds people together to allow them to use human and financial capital” (p. 3).
Section snippets
Program theory model and the national SS/HS initiative evaluation
The goal of the national cross-site evaluation of the SS/HS initiative is to assess the process and effectiveness of the initiative in improving outcomes and to develop understanding of implementation factors (i.e., process) that contribute to those improved outcomes. Yet several features of the SS/HS initiative combine to limit evaluation design options and contraindicate the use of an experimental or quasi-experimental design for the national evaluation. Most grant programs provide a single
Implementing the model in data definition and collection
Developing a hypothetical program model was the first step in the use of the model for evaluation; the next step was to define the model elements with collectible data. The national evaluation of the SS/HS initiative has faced numerous challenges in data definition and data collection. First, the environmental context varies noticeably across sites. It is essential for the national evaluation to consider contextual variation in assessing change in outcomes. Second, each community receiving a
Applying the program theory model to data analysis
The program theory model has also served as the basis for developing the analysis plan in the evaluation. As noted in Rossi et al. (2004), the articulation of the program theory must be in a form amenable to analysis. Therefore, for this evaluation, data analyses follow a staged analytic framework exploring distributional characteristics of each variable measure, followed by bivariate relationships to refine variable selection, followed by selected regressions, and finally multilevel modeling.
Lessons learned
Based on more than five years of experience, the national evaluation team of the SS/HS initiative has a practical understanding of the challenges that can be encountered during data collection and analysis for such a large, complex project. If not anticipated and managed effectively, these challenges can undermine efforts to accomplish the objectives of the evaluation.
Conclusion
The program theory model developed at the beginning of the national evaluation of the SS/HS initiative and refined over time has proven invaluable in guiding the evaluation design and analysis. In addition to its continued use in the ongoing evaluation of the grant program, preliminary findings from this evaluation effectively assist grantees and policymakers in making data-driven program and policy decisions and in communicating the initiative's successes to the public.
Acknowledgements
Funding for this study was provided by the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration within the U.S. Department of Health and Human Services. The content of this article represents the views of the authors and does not reflect the views of the Substance Abuse and Mental Health Services Administration or the U.S. Department of Health and Human Services.
Julia Rollison, M.P.H., currently serves as the Deputy Project Director at MANILA Consulting Group, Inc., for the national cross-site evaluation of the Safe Schools/Healthy Students initiative. She manages quantitative and qualitative data collection and coordinates data analysis efforts on key outcome measures, specifically those associated with GPRA requirements. She is responsible for ensuring the quality and integrity of data collected and creating and maintaining cross-site data
References (24)
- et al.
Program theory and logic model to address the co-occurrence of domestic violence and child maltreatment
Evaluation and Program Planning
(2010) Unfinished business: Causes and values
American Journal of Evaluation
(2001)- et al.
Mobilizing communities for teen pregnancy prevention: Associations between coalition characteristics and perceived accomplishments
Journal of Adolescent Health
(2005) - et al.
Putting program theory into operation
American Journal of Evaluation
(1999) The search for school climate: A review of the research
Review of Educational Research
(1982)- et al.
The community coalition action theory
- et al.
Readiness, functioning, and perceived effectiveness in community prevention coalitions: A study of communities that care
American Journal of Community Psychology
(2004) - et al.
Measuring, improving and sustaining healthy learning environments
Multilevel hierarchical modeling: What it can and cannot do
Technometrics
(2006)- et al.
School climate, academic performance, attendance, and dropout
(1989)
Major differences in the social climate of two high schools in the same town
Problem behavior and psychosocial development: A longitudinal study of youth
Cited by (3)
Challenges, Opportunities, and Methods for Large-Scale Evaluations
2018, Evaluation and the Health ProfessionsWorking better together: New approaches for understanding the value and challenges of organizational partnerships
2014, Health Promotion International
Julia Rollison, M.P.H., currently serves as the Deputy Project Director at MANILA Consulting Group, Inc., for the national cross-site evaluation of the Safe Schools/Healthy Students initiative. She manages quantitative and qualitative data collection and coordinates data analysis efforts on key outcome measures, specifically those associated with GPRA requirements. She is responsible for ensuring the quality and integrity of data collected and creating and maintaining cross-site data repositories. Ms. Rollison has expertise in program evaluation and performance measurement from past evaluations, such as the 5-year, multicity study on the outcomes of financing and organizing after-school activities by the Robert Wood Johnson Foundation, and the cross-site evaluations of 35 initiatives for SAMHSA's Fetal Alcohol Spectrum Disorders Center for Excellence.
Gary Hill, Ph.D., of MANILA Consulting Group, Inc., has nearly 40 years of increasing responsibility in the fields of health policy, program evaluation, project management, and the design and development of automated information systems. He currently serves as the Project Director for the Safe Schools/Healthy Students national cross-site evaluation. Previously, Dr. Hill was Division Director at a large federal government contracting firm, overseeing contracts providing, for example, program evaluation, cost–benefit analyses, and reports to Congress. Dr. Hill was the Project Director for the evaluation of the Robert Wood Johnson Foundation's After School Program. He was also the Project Director for the Department of State's International Demand Reduction Program, a $15 million, multiyear effort designed to coordinate the implementation and evaluation of prevention programs across the world.
Ping Yu, Ph.D., of Battelle Centers for Public Health Research and Evaluation, has more than 25 years of experience in community- and school-based substance abuse research, domestic and international program evaluation, and management of large-scale, multi-year, and multi-million-dollar projects. He currently serves as the Subcontract Director for Battelle on the national evaluation of the Safe Schools/Healthy Students initiative. Dr. Yu has managed a portfolio of research and evaluation projects worth more than $60 million across a wide range of government and private sector clients. He has designed and directed a number of large-scale studies that use both qualitative and quantitative methods to assess the effectiveness of community- and school-based prevention or treatment initiatives. Dr. Yu's contributions have received broad peer recognition in the form of published articles and participation in such forums as the National Committee on Asian and Pacific Islander Americans Health Research convened by the Director of Minority Health at the National Institutes of Health.
Stephen Murray, Ph.D., is the Subcontract Director at RMC Research Corporation for the Safe Schools/Healthy Students national evaluation and has served in this capacity for over 5 years. Dr. Murray has devoted his professional career to improving the quality of educational and social services through policy analysis, research and evaluation, and technical assistance. Currently, Dr. Murray is a key staff member on four large-scale studies that contribute to the field-testing of research-based programs to improve education. Prior to joining RMC, Dr. Murray directed evaluation projects for the Northwest Regional Educational Laboratory.
Danyelle Mannix, Ph.D., is a Senior Data Advisor for the Center for Mental Health Services at the Substance Abuse and Mental Health Services Administration in the U.S. Department of Health and Human Services. She is responsible for coordinating all data activities; identifying effective performance measures to demonstrate the efficiency and impact of CMHS’ nearly $1 billion grant portfolio; and ensuring that data-related products are suitable to inform policy and demonstrate accountability to the public. Previously, as a Special Expert in Evaluation, she was responsible for managing four national evaluations valued at over $40 million, i.e., the Safe Schools/Healthy Students, Garrett Lee Smith Suicide Prevention Grant Program, Native Aspirations, and National Child Trauma initiatives. In that capacity, she developed innovative strategies for integrating data collection and reporting activities into centralized data systems creating significant cost savings for the government and reducing burden on grantees. Ms. Mannix's significant contributions to SAMHSA have resulted in many job-related honors and awards.
Anne Mathews-Younes, Ed.D., is the Director of the Division of Prevention, Traumatic Stress, and Special Programs at the Substance Abuse and Mental Health Services Administration's Center for Mental Health Services. The Division is responsible for programs to prevent school violence and suicide; promote mental health and prevent mental and behavioral disorders; treat child trauma; and support disaster, terrorism, and bioterrorism preparedness and response; as well as coordinate, with the Federal Emergency Management Agency, the provision of crisis counseling services. Dr. Mathews-Younes is trained as an occupational therapist in addition to having received her doctorate in counseling and consulting psychology from Harvard University. She is a licensed psychologist and completed a master's degree in theological studies at Wesley Seminary in Washington, DC, where she is now pursing a doctoral degree in ministry in poverty studies.
Michael E. Wells, Ph.D., is a Grants Management Specialist and Team Lead at the U.S. Department of Education's Office of Safe and Drug Free Schools. He is responsible for the team that monitors the Safe Schools Healthy/Students Initiative and is the Project Lead on the Project SERV (School Emergency Response to Violence) grant program. Dr. Wells has worked in the fields of psychology and public education for over 30 years. He has had a private practice in psychotherapy, taught at the university level, has administered school safety and student services programs at the local and federal level, as well as establishing and directing the operation of public alternative school programs. Dr. Wells received his doctorate in counseling and educational development from the University of North Carolina and is licensed as a psychologist and a professional counselor.